I didn't try with chrony.  Hopefully this helps if you try NTP...

My ntp.conf includes:

# Server from shared memory provided by gpsd
server minpoll 4 maxpoll 4 prefer
fudge refid GPS stratum 4
fudge flag1 1

The flag1 parameter is to overcome the panic if the time is too far out of sync.

As mentioned above, the NTP shared memory only starts to get time from my GPS device after I wake it up with a client software such as gpsmon or cgps.  Otherwise, the system will sit there with no time source.  Starting gpsd in daemon mode fixes this, however, my indiserver indi_gpsd doesn't work then (if I could specify the indi_gpsd to connect to the network socket, rather than try to own the USB device, maybe that would work?)

Hope that helps.