I'm facing the same slow-down, however, it's on servers (1-host & 3 guest VMs). It appears to happen when you start a session, or multi-session, and let it sit for a while with no activity. Once you return, it's very sluggish and a lot of the time it will appear to be frozen. If you want for 1-3 minutes you can watch it paint the screen, trying to "catch-up" with the user.
It appear as-if the link between the session times-out .vs. keeping it alive with a "ping" to each other. So when the user returns to the session, it try's to re-initiate the connection - creating the lag. Problem is, the sessions returns, but it's noticeably slower than the initial session.