Uploaded image for project: 'Keycloak'
  1. Keycloak
  2. KEYCLOAK-6783

Cross-dc setup with JDG server on real network doesn't work

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 3.4.3.Final
    • Fix Version/s: 4.0.0.Beta1
    • Component/s: None
    • Labels:
      None
    • Docs QE Status:
      NEW
    • QE Status:
      NEW

      Description

      Cross-dc setup with the JDG server on real network doesn't work. What works and what we tested is:

      • Setup with infinispan-server-8.2.8 on "local" network (infinispan server bind on loopback address like "localhost" . Different infinispan servers running on the same laptop, but on various port offsets)
      • Setup with JDG server 7.1.0 on "local" network (JDG server bound on loopback address like "localhost" . Different JDG servers running on the same laptop, but on various port offsets)
      • Setup with infinispan-server-8.2.8 on "real" network (testing with infinispan hosts bound to real host with IP addresses like 192.168.0.1 )

      We didn't test with JDG server bind on "real" addresses and this is where the issue happens

      Easiest steps to reproduce:

      • Unpack JDG 7.1.0 server and configure clustered.xml like this:

                <replicated-cache-configuration name="sessions-cfg" mode="SYNC" start="EAGER" batching="false">
                    <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
                    <locking acquire-timeout="0" />
                    <!--<backups>
                        <backup site="site2" failure-policy="FAIL" strategy="SYNC" enabled="true">
                            <take-offline min-wait="60000" after-failures="3" />
                        </backup>
                    </backups>-->
                </replicated-cache-configuration>
         
                <replicated-cache name="work" configuration="sessions-cfg"/>
                <replicated-cache name="sessions" configuration="sessions-cfg"/>
                <replicated-cache name="clientSessions" configuration="sessions-cfg"/>
                <replicated-cache name="offlineSessions" configuration="sessions-cfg"/>
                <replicated-cache name="offlineClientSessions" configuration="sessions-cfg"/>
                <replicated-cache name="actionTokens" configuration="sessions-cfg"/>
                <replicated-cache name="loginFailures" configuration="sessions-cfg"/>
        

      • Start JDG server with command like this (assuming that 192.168.0.101 is the address of your laptop):

        ./standalone.sh -c clustered.xml -b 192.168.0.101 -Djava.net.preferIPv4Stack=true -Djboss.socket.binding.port-offset=1010 -Djboss.default.multicast.address=234.56.78.99 -Djboss.node.name=cache-server
        

      • Unpack keycloak server and configure standalone-ha.xml like described in the documentation
      • Run keycloak server with the command like:

        ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 -Djboss.site.name=site1 -Djboss.default.multicast.address=234.56.78.100 -Dremote.cache.port=12232 -Dremote.cache.host=192.168.0.101
        

      The keycloak server startup fails with the exception:

      Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=28 returned server error (status=0x84): org.infinispan.server.hotrod.RequestParsingException: Remote requests are allowed to protected caches only over loopback or if authorization is enabled. Do no send remote requests to cache '___script_cache'
      	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:350)
      	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:139)
      	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:125)
      	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
      	at org.infinispan.client.hotrod.impl.operations.PingOperation.execute(PingOperation.java:45)
      	at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.executeOperation(FaultTolerantPingOperation.java:31)
      	at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.executeOperation(FaultTolerantPingOperation.java:17)
      	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)
      	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:673)
      	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.resolveCompatibility(RemoteCacheImpl.java:734)
      	at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:666)
      	at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:536)
      	at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:532)
      	at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.init(RemoteCacheSessionsLoader.java:92)
      	at org.keycloak.models.sessions.infinispan.initializer.BaseCacheInitializer$1.run(BaseCacheInitializer.java:86)
      	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
      	at org.keycloak.models.sessions.infinispan.initializer.BaseCacheInitializer.getOrCreateInitializerState(BaseCacheInitializer.java:83)
      	at org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer.startLoading(InfinispanCacheInitializer.java:63)
      	at org.keycloak.models.sessions.infinispan.initializer.CacheInitializer.loadSessions(CacheInitializer.java:41)
      	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$7.run(InfinispanUserSessionProviderFactory.java:293)
      	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
      	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCache(InfinispanUserSessionProviderFactory.java:283)
      	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCaches(InfinispanUserSessionProviderFactory.java:275)
      	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.access$200(InfinispanUserSessionProviderFactory.java:64)
      	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.onEvent(InfinispanUserSessionProviderFactory.java:115)
      	at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:68)
      	at org.keycloak.services.resources.KeycloakApplication$2.run(KeycloakApplication.java:165)
      	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
      	at org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:158)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      	at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
      	... 33 more
      

      The JDG server has exception like:

      09:16:42,854 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-6-9) ISPN005003: Exception reported: org.infinispan.server.hotrod.RequestParsingException: Remote requests are allowed to protected caches only over loopback or if authorization is enabled. Do no send remote requests to cache '___script_cache'
      	at org.infinispan.server.hotrod.CacheDecodeContext.obtainCache(CacheDecodeContext.java:116)
      	at org.infinispan.server.hotrod.HotRodDecoder.decodeHeader(HotRodDecoder.java:145)
      	at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.java:93)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
      	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
      	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1018)
      	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)
      	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
      	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
      	at java.lang.Thread.run(Thread.java:748)
      
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  mposolda Marek Posolda
                  Reporter:
                  mposolda Marek Posolda
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: