We managed to solve connection problems.
In our case the problem was our custom ID server which we initially installed inside a virtualised server.
Since other servers are virtualised in the same environment, they share network interface and it seems this configuration interfered with ID server.
We have now moved ID server to physical computer and connection problems are gone.
Thanks to Conrad for great support.