Ah well, just as I feared. If I remember correctly back in the day Linux on ARM used to work in such a way that unaligned memory accesses (trying to access for example 32-bit data that is not in 32-bit aligned address) caused trap in the kernel and depending on options the kernel either fixed up the access or raised SIGBUS as it's usually a bug and slow in any case and older ARM processors didn't support unaligned access at all.

I made a short test program that does exactly this and it works on both my Raspberries (Pi 4 running 64-bit and Pi 3 running 32-bit RaspberryOS). It needs to be renamed to unaligned.c (forum doesn't allow .c files) and compiled with -O0 or the compiler optimizes the access away and just prints the value. It should print "Value 0x8010203". This might be totally red herring and have nothing to do with the issue, but checking that on the problematic platforms would at least narrow the search a bit.