Ian replied to the topic 'KStars memory leak and crash' in the forum. 6 months ago

Good idea. I've tested by

sudo systemctl stop astropanel.service indiwebmanager.service indi-mqtt.service
and verifying the processes stopped. Unfortunately, still seeing a leak.

Additionally, I built KStars and libindi from master on my x86_64 desktop and attempted to reproduce the memory leak without luck. Also built the 3.5.5 KStars and 1.9.2 libindi tags (versions astroberry is using), no dice.

Simple repro on astroberry:
1. backup and then rm -r ~/.local/share/kstars ~/.indi ~/.config/kstarsrc for a fresh start
2. launch KStars
3. launch Ekos
4. wizard local devices -> create profile -> add simulator telescope and CCD
5. start Ekos
6. go to guider tab -> loop
7. Check RES memory usage via htop, check again every few minutes and notice a steady increase

It doesn't appear to matter which tab it's left on. I've tested switching to the Scheduler tab in case it was due to images/charts updating but it still leaks.

Read More...

Ian replied to the topic 'KStars memory leak and crash' in the forum. 6 months ago

Update: I didn't end up imaging, however yesterday I tested a few cases:

1. Leaving KStars open without starting Ekos: no memory increase
2. Leaving KStars open with Ekos started, mount parked, guiding set to "loop": slow memory increase from 205MB to 355MB from 8:50AM to 10:50AM (1.25MB/min). No log messages about swig/python.
3. Building master (e7bc2c83c6d2975e049019d2b01fd16ff4e45625), stable-3.5.2, and 3.5.0 from source and repeating #2: same behavior.

This makes it seem like either a. it's not KStars, maybe an indi/astroberry issue or b. it's KStars, and either been around for a while or there's always been a slow leak and occasionally there's a larger leak that leads to the crash I experienced or c. it's KStars, and a recent indi/astroberry change has exposed an underlying issue.

Additonally, when closing KStars after #2 /usr/bin/indi_eqmod_telescope crashed. Looking back this happened twice on Dec 2nd as well. Not sure if it's related. Journalctl:

ec 05 10:50:13 astroberry systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Dec 05 10:50:13 astroberry systemd[1]: Started Process Core Dump (PID 20598/UID 0).
Dec 05 10:50:13 astroberry systemd-coredump[20599]: Process 17270 (indi_eqmod_tele) of user 1001 dumped core.                                                                                                      

                                                    Stack trace of thread 17270:
                                                    #0  0x00000000b63faf14 __GI_raise (libc.so.6)
                                                    #1  0x00000000b63e6230 __GI_abort (libc.so.6)
                                                    #2  0x00000000b66408d8 _ZN9__gnu_cxx27__verbose_terminate_handlerEv (libstdc++.so.6)                                                                           
Dec 05 10:50:13 astroberry systemd[1]: systemd-coredump@0-20598-0.service: Succeeded.

Core dump:
Core was generated by `indi_eqmod_telescope'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb63e6230 in __GI_abort () at abort.c:79
#2  0xb66408d8 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#3  0xb663e5b0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0xb663d56c in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)


Read More...

Ian replied to the topic 'KStars memory leak and crash' in the forum. 6 months ago

Jonathan, that's interesting. Sounds like it might be the same cause.

I was able to image last night without crashing, with KStars ending up at ~1GB reserved memory in the morning. I left Ekos on the Analyze tab, you're probably on to something with the tab being relevant. I think I left it on the Guiding tab last time, I'll try that tonight.

Downgrading might be possible if the packages are still in the apt cache, but I would be worried about settings not being compatible with older versions.

Read More...

Ian created a new topic ' KStars memory leak and crash' in the forum. 6 months ago

I recently updated all packages to the latest (available through astroberry) versions kstars-bleeding 3.5.5-1/libindi11.9.2-1. It's been a few months since I've updated (checking apt history: 3.4.3 -> 3.5.0 in January, 3.5.0 -> 3.5.2 in May, 3.5.2 -> 3.5.5 recently). Please let me know if there's any more info I can provide/debugging steps that would help.

I've noticed that the memory consumption of KStars slowly climbs throughout the night while imaging. Last night when checking on the mount just before the meridian flip was scheduled to start, I noticed KStars was closed while the mount was still tracking. Launching KStars notifies me that indi is running in the background and prompts to close the process to launch a fresh one. After launching KStars again, I set up a quick script to log memory usage to disk every 1m and noticed KStars steadily increased over the night from ~300MB to ~900MB reserved memory (no crash this time). Second highest memory usage was python (don't remember which script) @ ~60MB. In journalctl I see:

[color=#000000]Dec 02 21:31:25 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd [/color]
Dec 02 21:31:26 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:28 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:29 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:30 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:31 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:31:31 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:31:31 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:31:32 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:33 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:35 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:36 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:31:37 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd

repeatedly throughout the night.
usb 2-2 is the ASI120MM-S
, and from what I can tell this is "normal" behavior. The swig/python memory leak stands out however. Looking at the python processes running I see:
- /usr/bin/indi-mqtt.py
- /var/www/astropanel/astropanel.py
- /var/www/gpspanel/gpspanel.py
- /var/www/indiwebmanager/main.py
- /usr/bin/virtualgps.py

and it looks like the mqtt and web manager both talk with indi, maybe they're involved/triggering the leak somehow?

Backtrace (probably not helpful since it just crashed due to OOM):
[color=#000000]Core was generated by `kstars'. [/color]
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0xa3e81080 (LWP 1793))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb37ba230 in __GI_abort () at abort.c:79
#2  0xb380a50c in __libc_message (action=action@entry=do_abort, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:181                                                                                           
#3  0xb3811034 in malloc_printerr (str=<optimized out>) at malloc.c:5341
#4  0xb3812e50 in _int_free (av=<optimized out>, p=0x7e7c1b20, have_lock=0) at malloc.c:4258
#5  0xb55db970 in QVector<float>::reallocData(int, int, QFlags<QArrayData::AllocationOption>) () from /usr/lib/arm-linux-gnueabihf/libstellarsolver.so.1                                                            
#6  0xb55e3288 in InternalSextractorSolver::extractPartition(InternalSextractorSolver::ImageParams const&) () from /usr/lib/arm-linux-gnueabihf/libstellarsolver.so.1                                               
#7  0xb55ea5b0 in non-virtual thunk to QtConcurrent::RunFunctionTask<QList<FITSImage::Star> >::run() () from /usr/lib/arm-linux-gnueabihf/libstellarsolver.so.1                                                     
#8  0xb414df30 in ?? () from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
#9  0xb4156b58 in ?? () from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
#10 0xb4f06494 in start_thread (arg=0xa3e81080) at pthread_create.c:486
#11 0xb387a568 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6                                                                                                        
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

journalctl of crash (started KStars ~7:30pm, crashed ~9:54pm):
[color=#000000]Dec 02 21:52:25 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd [/color]
Dec 02 21:52:26 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:52:28 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:52:28 astroberry kernel: [color=#000000]usb 2-2: WARN: Max Exit Latency too large[/color][color=#000000] [/color]
Dec 02 21:52:28 astroberry kernel: [color=#000000]usb 2-2: Could not enable U2 link state, xHCI error -22.[/color][color=#000000] [/color]
Dec 02 21:52:28 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:52:28 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:52:28 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:52:29 astroberry kernel: usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Dec 02 21:52:30 astroberry systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Dec 02 21:52:30 astroberry systemd[1]: Started Process Core Dump (PID 5473/UID 0).
Dec 02 21:53:10 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:11 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:11 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:52 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:52 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:52 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:53:58 astroberry systemd-coredump[5474]: [color=#ff5454]Process 1782 ([/color][color=#ffffff]kstars[/color][color=#ff5454]) of user 1001 dumped core.[/color][color=#000000] [/color]
                                                   
                                                  [color=#ff5454]Stack trace of thread 1793:[/color][color=#000000] [/color]
                                                  [color=#ff5454]#0  0x00000000b37cef14 __GI_raise (libc.so.6)[/color][color=#000000] [/color]
                                                  [color=#ff5454]#1  0x00000000b37ba230 __GI_abort (libc.so.6)[/color][color=#000000] [/color]
                                                  [color=#ff5454]#2  0x00000000b380a50c __libc_message (libc.so.6)[/color][color=#000000] [/color]
                                                  [color=#ff5454]#3  0x00000000b3811034 malloc_printerr (libc.so.6)[/color][color=#000000] [/color]
                                                  [color=#ff5454]#4  0x00000000b3812e50 _int_free (libc.so.6)[/color][color=#000000] [/color]
                                                  [color=#ff5454]#5  0x00000000b55db970 _ZN7QVectorIfE11reallocDataEii6QFlagsIN10QArrayData16AllocationOptionEE (libstellarsolver.so.1)[/color][color=#000000] [/color]
Dec 02 21:53:59 astroberry systemd[1]: systemd-coredump@0-5473-0.service: Succeeded.
Dec 02 21:54:34 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:54:34 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:54:34 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:15 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:15 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:15 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:57 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:57 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:55:57 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:56:40 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:56:40 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:56:40 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:57:22 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:57:22 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:57:22 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:58:03 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:58:03 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:58:03 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.
Dec 02 21:58:45 astroberry python3[1047]: swig/python detected a memory leak of type 'INDI::BaseDevice::Properties *', no destructor found.

swig/python messages continue to print after kstars crashed, I'm assuming because indi is still running in the background.

Setup:
- Raspberry Pi 4 Model B/2GB running Astroberry
- ZWO ASI120MM-S (indi_asi_ccd)
- Sky-Watcher EQ6-R Pro (indi_eqmod_telescope)
- Olympus O-MD E-M10 Mark II (external intervalometer, not connected to Pi)

Imaging process:
- Go-to target
- Align -> Slew to target
- Adjust manually
- Align -> Sync
- Mount -> meridian flip @ 0.5h (scheduled to flip ~12am), HA limit 1.5h, park @ 4:10am
- Guide via EQMod Mount
- Start external intervalometer

Read More...

Hah, I was just typing up a reply actually.

I think ideally you would have each driver built as its own RPM package with a dependency on libraries if necessary.

The setup you have now looks great, are you running into any specific issues?

Read More...

@xsnrg that's awesome, thank you for doing this. I was considering creating Copr builds for it as well but hadn't gotten around to it yet.

If I could make a suggestion: is there a reason you're forking the repositories instead of setting up a remote for the spec (e.g. github.com/xsnrg/indi-rpm ) and pulling in the repo or tar as Source0? This way you build from upstream untouched, and don't need to maintain a fork.

As an example, one of my old Copr packages: github.com/ianhattendorf/gli-rpm

Read More...

That's unfortunate but I understand, thanks for everything you do! Just wanted to make sure it wasn't forgotten about, I should be able to just build the drivers I need manually for now.

Maybe just add a note to the page that Fedora 32+ are not currently supported instead of removing completely?

Read More...

Hi there,

Are the indi-bleeding builds for Fedora on OBS maintained by anyone here? It's currently not possible to install kstars/indi-bleeding on Fedora 32 due to a glibc update.

software.opensuse.org/download.html?proj...upinix-indi-bleeding

Read More...

Hi there,

Does someone here maintain the OBS builds for indi-bleeding? No builds exist for Fedora 32, and due to a glibc update Fedora 31 builds are not install-able.

software.opensuse.org/download.html?proj...upinix-indi-bleeding

Read More...