You definitely don't need to compile it yourself. `rpi-update` and its pre-compiled kernels is your friend and worked very well for me.
I explained and listed all of the steps I took here, hope it's OK to share;
How to connect EQ6-R Pro to AsiAIR (inc. other Raspberry PI solutions) through USB type B cable
> I found this document suggesting that 23d3 support was added in Linux 5.5
I was debugging the very same problem for days and looking for this very information. This line is the very reason why I am still sane (not completely though)!