This is very good point!
The leak is directly related to image files and their size. So there are two components to blame: CCD driver or EKOS.
I used indi_atik_ccd but the test should be run with other CCD drivers to narrow the search.

Please copy this simple script to /usr/local/bin/test_memleak
Then set the access rights:
Then add the script to EKOS CCD module when defining your session. Capture around 30 images and check the content of /tmp/memstat file , which contains memory available after each capture. Calculate average and compare it to a single FITS file (bin1 and bin2)

I have just tested it with ZWO ASI 120MM (file sizes: 2465 kB for BIN1 and 622 kB for BIN2)

30 x 1s BIN1
average memory drop: 2796 kB
standard deviation: 1101 kB

30 x 1s BIN2
average memory drop: 858 kB
standard deviation: 1126 kB

Apparently the memory leak is still related to file size and does not depend on CCD driver used.

Why do these leaks happen randomly with the same installation - no changes made?

Here goes an update on testing this nasty bug. When you set Upload to Local (instead of default Client), there is no memory leak. Even if you enable Summary Screen Preview and Use FITS Viewer options.
Make sure to set directory where your files should be written to. For whatever reason the parameter is called Remote (it is on the right side of Upload drop-down control) in Ekos CCD tab.
Stay tuned

Got a screen shot?

Look the screenshot above and set the options just below Directory: Upload to Local, Remote to directory where you want the files to be saved.
This way files will be saved only on the machine where INDI server is running.

Interesting; you might be onto something as I have mine set to "client" as well and have been experiencing the crash, as you know....



Jim

I think I nailed it! So the actual memory leak is related to FITS Viewer.
1. If you set Upload to Local, FITS Viewer does not render preview images (this is a feature) - no memory leak
2. If you set Upload to Client, you can enable/disable image preview in FITS Viewer in separate window (KStars / Settings / Configure KStars / FITS / Use FITS Viewer )
a) FITS Viewer disabled - confirmed memory leak at image file size for every captured image
b) FITS Viewer enabled - possible minor memory leak BUT memory is released right after you close FITS Viewer

I will dive into the code after this finding so a persistent solution is developped.
In the meantime... If you face a memory leak bug and you use Upload to Client option in Ekos, you should ENABLE FITS Viewer (KStars / Settings / Configure KStars / FITS / Use FITS Viewer ) and close the viewer window from time to time.
Otherwise use Upload to Local, which will make you immune to this bug.

Thanks! I actually never ran it without FITS viewer enabled... I'll get right to it.

I am not sure this is the ENTIRE problem, at least on Ubuntu MATE 18.04. This is the memstat file with FITS viewer ON and rendering preview (latest capture ONLY):

GNU nano 2.9.3 /tmp/memstat

1914512
1902872
1909940
1905392
1898224
1895228
1894848
1902876
1896328
1897328

[ line 31/31 (100%), col 1/1 (100%), char 240/240 (100%) ]
^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos
^X Exit ^R Read File ^\ Replace ^U Uncut Text^T To Spell ^_ Go To Line

As I reported before, memory usage remains steady at ~ 2GB as it did the previous night when I captured ~15GB worth of frames at (40 MB each).

HOWEVER, I can reproduce your observations re memory leak when the FITS viewer is OFF:

This video shows what I found when I tested this:

www.dropbox.com/s/vtw8varxtzcwzn7/MemoryLeak2.mkv?dl=0

Memory, including Swap space, fills up and eventually Capture freezes, but Kstars does not crash.

BUT, when I turn FITS viewer on there is NO memory leak and Capture proceeds normally. Memory is not filling up (watch the System Monitor in the task bar, second from left is RAM usage, fourth from the left is Swap space used).

www.dropbox.com/s/jf64d6ajm5t0jso/MemoryLeak3.mkv?dl=0

But when I stop the Capture sequence, THEN Kstars crashes.

I hope this will be useful to you finding the cause for what is going on in the code.

Jo

PS: Forgot to mention (but it shows in the videos): Upload was ALWAYS set to Client only). So this is independent of the Local Upload setting.

It would be useful to run memstat one more time AFTER fitsviewer is closed. I have noted that it recovers locked memory.
In your test total leak is 17 184 kB, average per file is 693 kB per capture. This might confirm possible minor leak as already noted.
What is your FITS file size?

PS. If it was always set to Upload: Client, the result IS dependent on the setting. Contrary of you saying "it is independent". Please test Upload: Local to confirm this

Please check out this video and let me know whether I tested all the combinations you wanted:

www.dropbox.com/s/1k7jqgxd7lcd8tp/MemoryLeak4.mkv?dl=0

Interestingly, when I set Upload:Local, the FITS viewer does not come on, although it is selected in the FITS options in KStars.
When FITS viewer is on (Upload:Client setting) there this no memory leak I can discern (memory usages if fluctuating around pretty much the same values), but when I disable it then the leak occurs. When I reengage FITS viewer the leak stops, but the memory is not released.
Either way, when I stop Capture under either setting then KStars crashes.

Please let me know if overlooked anything or misinterpret things.

Jo

Forgot to mention: File size 40 MB each.