I have a few updates. I haven't been able to change or add a port number, and 443 won't work. Regardless, I took another look at the logging on the system that is running the host that gets the error, and I see that there is an error in the log file:
a) Error code: 35
b) IP address: 127.0.0.1
c) Description: Password is incorrect or error occurs.
I can tell you with certainty that all the passwords are correct. In fact, there are no passwords to enter by this point, as I am already logged into the RU server by the time I get to this point.
Last night, I was also able to get the customer server security authorization error for some other hosts (the two that access the RU server via the intranet), but I don't believe that it was caused by the same problem. On the RU server system, the RU Server user interface was unable to connect to the server and kept on reporting access denied, even over a named pipe connection. Rebooting the system didn't fix anything. I ended up deleting the HKEY_LOCAL_MACHINE\SOFTWARE\Usoris\Remote Utilities\MiniInternetId Registry key, which caused settings to be reverted to their defaults, and that fixed the problem with being able to connect to the RU Server from the UI (note that address book information appears to be stored elsewhere, so that wasn't lost). But, now, the RU server had a new public and private key, which didn't match that used on the hosts. So, I had to go to each host that uses the RU server for CSS and login again, which displayed a warning message that the keys didn't match, etc, etc. Anyway, during this whole sequence, sometimes it would report CSS authorization error, and just updating the key wasn't enough. I ended up deleting users and groups from the hosts, turning off CSS, turning it back on, and then adding users and groups again. That was enough to get past the CSS error. I also tried this on the problematic host, but it didn't help, and I continue to get the CSS error, which makes me think that it is caused by something else.
I can't change the port, as there are multiple hosts, besides the three that I mentioned, that will stop being able to communicate with the server if I do that. Also, getting someone in IT to enable port forwarding for an additional port is not realistic at this point. Regardless, it's unclear to me why using a different port number would make a difference anyway. The host for which the problem occurs can communicate just fine with the RU server over the existing port. I can connect to this host just fine if I use any other authentication protocol besides CSS. The host clearly can establish some communication with the server using CSS, because otherwise, I wouldn't have been able to add users/groups to it under the host's CSS settings. In addition, if I run the viewer on the same system running the host, I can successfully login to the RU server using CSS over the public Internet. So, the issue appears to be specific to the communication between viewer and host, via the RU server, during the authentication phase, but only when CSS is used.
Edited:Aaron Levinson - Nov 25, 2019 1:55:16 pm EST
I already tried both :-) . It didn't make a difference. I think I had read on another post that "auto" chooses custom server security first if that protocol is enabled on the host. I tried a bunch of things to isolate the issue when I investigated it, and the only thing that I identified that was different about the setup is that the host in question connects to the RU server via the Internet, as opposed to the intranet. But, that doesn't mean that's the cause of the issue.
I already know that my RU server is accessible from the public Internet, and I don't need to use canyouseeme.org to check. I thought this would have been clear from my first post on this topic, since I described successfully connecting a remote host to the server over the public Internet, but I also successfully connected to it from a system running the RU viewer application on the public Internet a few days ago. I had no problem logging into it using the custom server security protocol from the viewer, and I also had no problem connecting to hosts that are connected to the server via either the intranet and the Internet.
I have the latest software (as of 11/21/2019) installed on several systems. RU Server on one system, RU Host on three systems (including the system running the RU Server itself), and the viewer installed on a few systems as well. I have Custom Server Security enabled on the RU Server and on the hosts as well, and this is the only authorization protocol enabled on the hosts.
I have no problem logging into two of the hosts, but for one of them, no matter what I do, I always get the "Custom Server Security Authorization Error" message. The main difference between the three hosts is that, for the two that I can login to, they are on the same network as the RU Server and use an Intranet IP address to connect to it for authorization purposes and Internet ID purposes. For the host for which I get the error, it connects to the RU Server across the Internet and uses its public-facing IP address for both authorization and Internet ID.
I can connect to the problematic host without any issue if I use the Windows user authentication protocol, and I assume that it would work with the single password and RU user auths, although I haven't tried those specifically. So, the issue appears to be specific to the custom server authentication protocol.
I even tried running the viewer from the host itself, connecting by either IP address or Internet ID, but I got the error in all cases.
I examined the viewer and host log files, but there is nothing that is obvious in there, to me, that points to the cause of the problem. For the host log file, nothing is added when the error occurs.
This seems like a bug to me, and I setup all the hosts the same way except for using the public IP address instead of the internal IP address. In addition, it is clearly able to access the RU server for custom server security authentication purposes over the public Internet, because otherwise, I would not have been able to add users and groups to the host in the first place. I following the instructions at https://www.remoteutilities.com/support/docs/setting-up-auth-server/ for setting all of this up.