Performance Center Virtual Load Generators are pausing for up to 40 seconds during load testing

Currently using a Performance Center install (version 12.50) with a mix of physical and virtual load generators.  

Over the last few months, the team have been reporting blips or spikes in response times that are intermittent on the virtual load generators.  During further investigation, the spikes in response times are not caused by the AUT, but is within the virtual machines somewhere.  Wireshark traces show the Virtual machine communicating with the F5 load balancer but then it stops sending or receiving traffic causing a severe spike in response time.  The transaction does not fail and then the virtual machines recovers and the test continues.

This issue is now fairly repeatable but the root cause of the problem still cannot be found.  Currently speaking with HP on the issue but again no solution is forthcoming yet.  We do not see this issue with the physical load generators but as we have quite a few virtual machines it would be good to get this resolved. 

There seems to be concern around clock drift with virtual machines but I don't know if this is the case here as timings appear to be correct within the machine when the issue occurs.

I just wondered if anyone had experienced similar issues?  I can't find much in the way of similar issues through doing online research.