Hi Adrien.
No sorry I don't have a solution, I only want to tell you that I am succesfully running RPI the way you describe.
I started with RP3 as a hotspot running only indiserver with Kstars/Ekos running on my laptop connecting via wifi.
Connected devices on the RP were my mount, CMOS-camera and a Polemaster camera
This year I upgraded to a RP4 with 4GB RAM. I then moved all software to RP including
Kstars/Ekos. Still running as a hotspot I only use a simple VNC-viewer to connect.
I connect my CMOS-cameras to USB3 and the mount and Polemaster to USB2.
Works with no problem, I am often out in -15 degrees Celsius or colder, sitting in my car controlling everything
and no problem.
I know stable power is important. And have you tested an external wifi-dongle?
So don't give up, it should work. Sorry that I can't help you more.