When using a remote jndi connection to AS and a custom security-realm for it that is tied to a custom login module the authentication only works when packaging the class in a jar and installing it on jboss_install_dir/modules/org/jboss/as/remoting/main. It should be loadable from ear also.
More details here: https://community.jboss.org/thread/195501