On my side it is long but regular (except for the first acquisition that car take a little bit more, about 20% more time). I do not have, as you, changes in the acquisition time or endless loop.
I didn't resolved this problem, for the moment this is "almost sufficient" for my projet that just need 5 to 10 Hz acquisitions and I reached 3 Hz.
I'm working on a python library that call a c++ client that just take an image and send it to the python software. I hope it will help.