I've noticed sometime in the last few updates, that when I go to Full Control clients, that about 10-12 seconds is wasted on a failed first authorization attempt and the 2nd time usually succeeds. On occasion, I've had it fail 5 times and then closing and reopening the Viewer got it to connect again, I think on the first try. It didn't use to do this regularly.
Please, make sure that you have the same version (the latest one) on both ends, Viewer and Host. If you do, then these issues may occur if the Host is temporarily briefly offline (which is not immediately reflected by its status in the address book since there's a status check with 1-minute interval).