Keystore Errors after Upgrading to Rollup 4

First, I generated a CA and certificates and placed them in a common location on the server. I used /opt/fuse/keystore/server-chain.jks. Make sure the keystore is also added as a trustStore to the JVM in setenv: export EXTRA_JAVA_OPTS= "-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass" I started with JBoss Fuse 6.2.1 Rollup 1 (621090): > fabric:create --wait- for -provisioning --zookeeper-password admin -- new -user admin -- new -user-password admin -- new -user-role Administator --global-resolver manualip --resolver manualip --manual-ip node1.redhat.com Add jasypt support: > profile-edit --feature jasypt-encryption default Set a default encryption password: > crypt-password-set gingerbreadman Encrypt the keystore password: > encrypt-message changepass Encrypting message changepass Using algorithm PBEWithMD5AndDES and password gingerbreadman Result: LdWR1f3SFKB1hojHK8Bz2ba4IjruBOU7 Create a new version for SSL configuration: > version-create 1.1 Add the SSL properties to pax web: > profile-edit --resource org.ops4j.pax.web.properties default 1.1 # # Copyright 2005-2014 Red Hat, Inc. # # Red Hat licenses this file to you under the Apache License, version # 2.0 (the "License" ); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http: //www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. # org.osgi.service.http.port=${port:8181,8282} javax.servlet.context.tempdir=${runtime.data}/pax-web-jsp org.ops4j.pax.web.config.url=profile:jetty.xml org.ops4j.pax.web.config.checksum=${checksum:profile\:jetty.xml} org.osgi.service.http.enabled= false org.osgi.service.http.secure.enabled= true # Specify a range here, so multiple containers can use the profile org.osgi.service.http.port.secure=${port:8443,8543} org.ops4j.pax.web.ssl.keystore=/opt/fuse/keystore/server-chain.jks org.ops4j.pax.web.ssl.password=${crypt:LdWR1f3SFKB1hojHK8Bz2ba4IjruBOU7} org.ops4j.pax.web.ssl.keypassword=${crypt:LdWR1f3SFKB1hojHK8Bz2ba4IjruBOU7} Upgrade container and test SSL connectivity: > container-upgrade 1.1 node1 (verify ssl connections are working with a browser connection to the container on 8443 or configured port) Create a new version for Rollup 4 patch: > version-create 1.2 Created version: 1.2 as copy of: 1.1 Patch the new version: > patch:add file: ///opt/fuse/patch/jboss-fuse-full-6.2.1.redhat-159.zip > patch:fabric-install --username admin --password admin --upload --version 1.3 jboss-fuse-full-6.2.1.redhat-159 (restore overwritten configurations) > profile-edit --resource org.ops4j.pax.web.properties default 1.2 # # Copyright 2005-2014 Red Hat, Inc. # # Red Hat licenses this file to you under the Apache License, version # 2.0 (the "License" ); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http: //www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. # org.osgi.service.http.port=${port:8181,8282} javax.servlet.context.tempdir=${runtime.data}/pax-web-jsp org.ops4j.pax.web.config.url=profile:jetty.xml org.ops4j.pax.web.config.checksum=${checksum:profile\:jetty.xml} org.osgi.service.http.enabled= false org.osgi.service.http.secure.enabled= true # Specify a range here, so multiple containers can use the profile org.osgi.service.http.port.secure=${port:8443,8543} org.ops4j.pax.web.ssl.keystore=/opt/fuse/keystore/server-chain.jks org.ops4j.pax.web.ssl.password=${crypt:LdWR1f3SFKB1hojHK8Bz2ba4IjruBOU7} org.ops4j.pax.web.ssl.keypassword=${crypt:LdWR1f3SFKB1hojHK8Bz2ba4IjruBOU7} Upgrade container: > container-upgrade 1.2 node1 > shutdown (container needs restart) Upon subsequent restarts, the following log error is observed: 2016-11-08 15:47:38,508 | WARN | pool-3-thread-1 | AbstractLifeCycle | 96 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.19.v20160209 | FAILED SslContextFactory@54ae12e3(/opt/fuse/keystore/server-chain.jks,/opt/fuse/keystore/server-chain.jks): java.io.IOException: Keystore was tampered with, or password was incorrect java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)[:1.7.0_111] at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)[:1.7.0_111] at java.security.KeyStore.load(KeyStore.java:1226)[:1.7.0_111] at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:55)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyStore(SslContextFactory.java:1052)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1012)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:264)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:612)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.startConnector(ServerControllerImpl.java:569)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.start(ServerControllerImpl.java:531)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.start(ServerControllerImpl.java:69)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Unconfigured.configure(ServerControllerImpl.java:686)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.configure(ServerControllerImpl.java:85)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:341)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at org.ops4j.pax.web.service.internal.Activator$2.run(Activator.java:276)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_111] at java.lang. Thread .run( Thread .java:745)[:1.7.0_111] Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)[:1.7.0_111] ... 21 more 2016-11-08 15:47:38,513 | WARN | pool-3-thread-1 | AbstractLifeCycle | 96 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.19.v20160209 | FAILED SslSelectChannelConnector@0.0.0.0:8443: java.io.IOException: Keystore was tampered with, or password was incorrect java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)[:1.7.0_111] at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)[:1.7.0_111] at java.security.KeyStore.load(KeyStore.java:1226)[:1.7.0_111] at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:55)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyStore(SslContextFactory.java:1052)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1012)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:264)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:612)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.startConnector(ServerControllerImpl.java:569)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.start(ServerControllerImpl.java:531)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.start(ServerControllerImpl.java:69)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Unconfigured.configure(ServerControllerImpl.java:686)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.configure(ServerControllerImpl.java:85)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:341)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at org.ops4j.pax.web.service.internal.Activator$2.run(Activator.java:276)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_111] at java.lang. Thread .run( Thread .java:745)[:1.7.0_111] Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)[:1.7.0_111] ... 21 more 2016-11-08 15:47:38,522 | WARN | pool-3-thread-1 | ServerControllerImpl | 104 - org.ops4j.pax.web.pax-web-jetty - 3.2.9 | Http connector will not be started java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)[:1.7.0_111] at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)[:1.7.0_111] at java.security.KeyStore.load(KeyStore.java:1226)[:1.7.0_111] at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:55)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyStore(SslContextFactory.java:1052)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1012)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:264)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:612)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)[96:org.eclipse.jetty.aggregate.jetty-all-server:8.1.19.v20160209] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.startConnector(ServerControllerImpl.java:569)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Stopped.start(ServerControllerImpl.java:531)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.start(ServerControllerImpl.java:69)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Unconfigured.configure(ServerControllerImpl.java:686)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.configure(ServerControllerImpl.java:85)[104:org.ops4j.pax.web.pax-web-jetty:3.2.9] at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:341)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at org.ops4j.pax.web.service.internal.Activator$2.run(Activator.java:276)[103:org.ops4j.pax.web.pax-web-runtime:3.2.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_111] at java.lang. Thread .run( Thread .java:745)[:1.7.0_111] Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)[:1.7.0_111] ... 21 more 2016-11-08 15:47:38,524 | INFO | pool-3-thread-1 | HttpServiceFactoryImpl | 103 - org.ops4j.pax.web.pax-web-runtime - 3.2.9 | Binding bundle: [org.apache.cxf.cxf-rt-transports-http [205]] to http service