Thank you all for your replies and guidance on troubleshooting. I can confidently say now that it's a power issue on the USB. As Kevin was suggesting, I checked the logs and found indeed low voltage warnings. I ordered a good rugged powered USB hub that will hopefully solve my problems.
In the meantime, I have two "solutions". One of them is to keep rebooting and replugging stuff until it works. I boot the Pi, and once I see it in the network I plug devices one by one, slowly, keeping an eye on the ping. Whenever the ping drops (as in it stops ponging back), I just disconnect stuff and reboot the Pi to try again. About 25% of the time it works, and when it does it's good for hours.
The other solution is to not plug the filter wheel and stick to luminance for now. I noticed the rig is a lot more stable (about 80% success) when there is no filter wheel.
You can guess by reading my "solutions" how frustrated I was when trying to make things work! It worked fine back in the good old days where I had a DSLR, an autoguider and a mount, so I ended up relying a lot on that Pi. But now it's pretty much unusable until I get that hub (hopefully).
For information, it's a Pi4B with 4GB RAM. I also tried with a Pi4B 8GB (by swapping the SD card) but did not notice any improvement.

Read More...