I know this is a few months old, but I'm having the exact same problem. I'm connected to a Celestron CGEM with the "SkyLink" WiFi adapter plugged into the Aux port and using the INDI "AUX" driver. Everything connects great and the mount moves in the right directions when using the virtual hand controller. I can sync to a star and it shows the correct RA/DEC values, but as soon as I tell it to goto a nearby star (only a short distance away), the mount shoots off in the opposite direction and I have to mash the STOP button to stop the camera crashing into the mount.
I've gone back to connecting through the hand controller with a USB/Serial ST4 cable which works great. Downside is you have to align with the HC first before using INDI.
One thing I did catch out of the corner of my eye, was when doing a goto, the virtual hand controller flashes "error" for a fraction of a second and then immediately changes to "slewing"