I have the latest software (as of 11/21/2019) installed on several systems. RU Server on one system, RU Host on three systems (including the system running the RU Server itself), and the viewer installed on a few systems as well. I have Custom Server Security enabled on the RU Server and on the hosts as well, and this is the only authorization protocol enabled on the hosts.
I have no problem logging into two of the hosts, but for one of them, no matter what I do, I always get the "Custom Server Security Authorization Error" message. The main difference between the three hosts is that, for the two that I can login to, they are on the same network as the RU Server and use an Intranet IP address to connect to it for authorization purposes and Internet ID purposes. For the host for which I get the error, it connects to the RU Server across the Internet and uses its public-facing IP address for both authorization and Internet ID.
I can connect to the problematic host without any issue if I use the Windows user authentication protocol, and I assume that it would work with the single password and RU user auths, although I haven't tried those specifically. So, the issue appears to be specific to the custom server authentication protocol.
I even tried running the viewer from the host itself, connecting by either IP address or Internet ID, but I got the error in all cases.
I examined the viewer and host log files, but there is nothing that is obvious in there, to me, that points to the cause of the problem. For the host log file, nothing is added when the error occurs.
This seems like a bug to me, and I setup all the hosts the same way except for using the public IP address instead of the internal IP address. In addition, it is clearly able to access the RU server for custom server security authentication purposes over the public Internet, because otherwise, I would not have been able to add users and groups to the host in the first place. I following the instructions at https://www.remoteutilities.com/support/docs/setting-up-auth-server/
for setting all of this up.