Hello Adam,
The "Advanced Hint Window" does not show while hovering over an "online" connection unfortunately.
Yes, unfortunately, if the connection is simply "Online" you still would not be able to see the "preview" on the Advanced Hint window - in order to see it, you need to log in to the remote Host. You can also log in without starting a remote session right away - simply right-click on the preferred connection and select "Logon" or navigate to the
Connection tab of the Viewer's ribbon toolbar and select "Log on" as described
here. This will allow you to see preview of the remote screen in both - address book and in the Advanced Hint window without starting a remote session.
Hope that helps.