We managed to solve connection problems. In our case the problem was our custom ID server which we initially installed inside a virtualised server. Since other servers are virtualised in the same environment, they share network interface and it seems this configuration interfered with ID server.
We have now moved ID server to physical computer and connection problems are gone.
Unfortunately, no such thing is possible, it is a live server.
Only thing I noticed is that server has three network interfaces. I do select one in remote utilities but could it be this is confusing host software and is trying to switch between interfaces which is causing issues then?