I too am seeing this error and would like to know a fix and what I can provide to help resolve.

I am running astroberry.io on RPi 4 with 4GB on a 128 GB SD Card.  I have networking enabled to allow remote access for SMB and VNC.

I saw this in the version since Jan 2021, and it used to have a popup dialog box as shown in the attached image.

I upgraded in August 2021 and now the error is no longer in the dialog but is in the message log on the bottom of the screen.  It causes "reset of the file counter to be xyz_<datetime>_001.fits for all files after the error".  It has Status for all jobs shown as "Invalid" when it was already loaded and running for several of the images.

Like the original poster, I create jobs with many targets (> 10 per night), but this is not a requirement.

I had thought this was related to targets in the job "dropping below the horizon" as the job typically runs for 8 hrs (all night) and the first targets drop below 0deg, but this is not a required condition.

I could not find the pic, but will certainly get the error tonight and will add to the post.
 

Hi Scott

Sorry to hear you might be experiencing the same issue - I'm not sure we are both seeing the exact same issue but it sounds like we possibly are.

I was waiting to hook up an external drive to see if this might be due to reading/writing to the SD card. Sadly it appears that this is not the case. I am easily able to reproduce the issue by running a schedule specifying unlimited images (I set the exposure length to 1s). I'm using the SX camera driver with an all sky cam so my schedule is far simpler than yours by the sound of it. I got the attached issue after 322 1s subs this time. The actual number of subs before the issue varies for me.

I'm happy to put some time/effort into trying to solve this.  I will confirm that I have the latest version (might switch to nightly builds) and reconfirm the issue.  Then I guess the next step might be to try and run kstars in a debug session which I have read about elsewhere in the forum.

Cheers
Iain

I agree. it is not due to number of targets as I will see this with only 6 targets in the sequence. I can be certain I will have this problem 2-3 times in the night so surprised not many others have mentioned this. It will be cloudy this week, so will do some research on collecting logs and run in debug mode and try to isolate the issue. part of my problem is the dialog error is clearly not the real problem but rather a symptom of something else. some of the ideas I have had are related to SD card read fails, or there is a background loop that is reviewing the targets and updating info like date/time for processing and angle in sky and it must fail the read in the process and that results in fail.

It would be greatly appreciated if the root of this issue is identified since apparently it affects multiple users.

Hi Jasem

Thanks for your interest in this issue.  I was able to simplify my setup last night and still produce the problem.  I removed the Telescope Simulator from my profile and now I am just running the SX driver.  I can reliably reproduce the problem in about 5 minutes with 1s subs.  I know you looked at this before and were unable to reproduce the issue.  I am thinking it might be to do with the number of files in the target directory growing large enough for a problem to appear when resolving the subsequent file number (that's just a hunch though).

It would be interesting to know if in Scott's setup he has large numbers (100's) of files in the target image directories?  I tend to be able to produce between 300-500 images before the issue appears.

I still need to:

1. Update my KStars to ensure latest version (as Scott says he does not get the dialog but I do).
2. Run a gdb debug session of kstars and reproduce the issue.

Cheers
Iain

Thank you. Looking forward to the findings.

I could reproduce the error using the 1 sec image as mentioned above.

1)  setup camera collection as expose 1 sec with 50 count, with filter/gain/offset as needed, give name (Lseq1sec50.....esq)
2)  setup schedule on target (Dubhe) with no track, no focus, no align, no guide using sequence from (1).  setup 20 targets with names like (Dub-1, Dub-2, Dub-3, ... as repeats)

a)  When it runs I get to Dubhe-7 set (about 350-400 images collected).
b)  Fail shows up as "invalid" and issue highlighted above.  Image attached shows sequence in middle of fail.  Second image shows (a) the list of files in D-7 set and you will see gets multiple files named file_xxxxxx_039.fits (this was the beginning of the fail).  I "reset" the camera without stopping the sequence or capture because failed on download of image. (these are the xxxx_040.fits files).  after it gets all the way through the 50 files in the set, the sequencer notes complete and all other items are "invalid", so does shutdown process and closes all items and stops.
c)  Attempt to "restart/reconnect" ekos without closing application will result in eventual kstars application crash. I restarted ekos to check the camera and crashed.  Restart of kstars, restart of ekos after crash will work OK -  this is what I normally do when I get invalid as other UI features are broken if you do not restart.
d)  attached are esl and esq files.

   
 

 

Repeat the above sequence -- fails at section 8 file 39-40.

Collect and attach Log file!  Below / zipped.

 

recapture the head of log for restart to show initialization as missing from log above

[2021-09-14T11:10:50.414 CDT INFO ][ org.kde.kstars] - Welcome to KStars 3.5.4 Stable
[2021-09-14T11:10:50.415 CDT INFO ][ org.kde.kstars] - Build: 2021-07-15T21:15:09Z
[2021-09-14T11:10:50.416 CDT INFO ][ org.kde.kstars] - OS: "raspbian"
[2021-09-14T11:10:50.417 CDT INFO ][ org.kde.kstars] - API: "arm-little_endian-ilp32-eabi-hardfloat"
[2021-09-14T11:10:50.417 CDT INFO ][ org.kde.kstars] - Arch: "arm"
[2021-09-14T11:10:50.418 CDT INFO ][ org.kde.kstars] - Kernel Type: "linux"
[2021-09-14T11:10:50.419 CDT INFO ][ org.kde.kstars] - Kernel Version: "5.10.60-v7l+"
[2021-09-14T11:10:50.419 CDT INFO ][ org.kde.kstars] - Qt Version: 5.11.3
[2021-09-14T11:10:50.438 CDT INFO ][ org.kde.kstars] - KStars is started in paused state.
[2021-09-14T11:10:50.712 CDT DEBG ][ org.kde.kstars] - Opened the User DB. Ready.
[2021-09-14T11:10:51.228 CDT INFO ][ org.kde.kstars] - Processing "unnamedstars.dat" , HTMesh Level 3
[2021-09-14T11:10:51.228 CDT INFO ][ org.kde.kstars] - Sky Mesh Size: 512
[2021-09-14T11:10:51.316 CDT INFO ][ org.kde.kstars] - Loaded DSO catalog file: "unnamedstars.dat"
[2021-09-14T11:10:51.317 CDT INFO ][ org.kde.kstars] - Processing "deepstars.dat" , HTMesh Level 3
[2021-09-14T11:10:51.318 CDT INFO ][ org.kde.kstars] - Sky Mesh Size: 512
[2021-09-14T11:10:51.318 CDT INFO ][ org.kde.kstars] - Loaded DSO catalog file: "deepstars.dat"
[2021-09-14T11:10:52.104 CDT WARN ][ org.kde.kstars] - "Star HD20794 not found."
[2021-09-14T11:10:52.140 CDT WARN ][ org.kde.kstars] - "Star HD98230 not found."
[2021-09-14T11:10:52.179 CDT INFO ][ org.kde.kstars] - Loaded DSO catalogs.
[2021-09-14T11:10:52.633 CDT DEBG ][ default] - File opened: "/home/astroberry/.local/share/kstars/comets.dat"
[2021-09-14T11:10:53.389 CDT WARN ][ org.kde.kstars] - Current icon theme is "PiXflat"
[2021-09-14T11:10:53.434 CDT DEBG ][ org.kde.kstars] - Paths to color scheme : ("/usr/share/kstars/themes")
[2021-09-14T11:10:54.404 CDT DEBG ][ org.kde.kstars] - Reporting new timestep value: 60
[2021-09-14T11:10:54.814 CDT DEBG ][ qt5ct] - D-Bus global menu: no
[2021-09-14T11:10:54.839 CDT CRIT ][ default] - Shortcut for action "get_data" "Download New Data..." set with QAction::setShortcut()! Use KActionCollection::setDefaultShortcut(s) instead.
[2021-09-14T11:10:54.985 CDT DEBG ][ org.kde.kstars] - Daylight Saving Time active
[2021-09-14T11:10:54.987 CDT DEBG ][ org.kde.kstars] - Next Daylight Savings Time change (Local Time): "Sun Nov 7 02:00:00 2021 GMT"
[2021-09-14T11:10:54.988 CDT DEBG ][ org.kde.kstars] - Next Daylight Savings Time change (UTC): "Sun Nov 7 07:00:00 2021 GMT"
[2021-09-14T11:10:55.617 CDT DEBG ][ org.kde.kstars] - Date/Time is: "Tue Sep 14 16:10:50 2021 GMT"
[2021-09-14T11:10:55.617 CDT DEBG ][ org.kde.kstars] - Location: "Home, Texas, USA"
[2021-09-14T11:10:55.618 CDT DEBG ][ org.kde.kstars] - TZ0: -6 TZ: -5
[2021-09-14T11:10:55.619 CDT WARN ][ org.kde.kstars] - Current icon theme is "breeze-dark"
[2021-09-14T11:10:55.714 CDT DEBG ][ org.kde.kstars] - "Black Body" :: "/usr/share/kstars/themes/blackbody.colors"
[2021-09-14T11:10:56.094 CDT DEBG ][ default] - glibc >= 2.1 detected. Using GNU extension sincos()
[2021-09-14T11:10:56.114 CDT DEBG ][ default] - static bool LibSecretKeyring::findPassword(const QString&, const QString&, QKeychain::JobPrivate*)
[2021-09-14T11:10:57.449 CDT DEBG ][ default] - File opened: "/home/astroberry/.local/share/kstars/comets.dat"
[2021-09-14T11:10:59.914 CDT DEBG ][ default] - File opened: "/home/astroberry/.local/share/kstars/asteroids.dat"

Hi Scott

I did not dig through your logs yet - I will - but I did find this in mine from last night's crash.

[2021-09-14T04:26:06.200 BST WARN ][                       default] - QProcessPrivate::createPipe: Cannot create pipe 0x6e95414: Too many open files

In context below:

[2021-09-14T04:23:03.151 BST DEBG ][ org.kde.kstars.ekos.scheduler] - Frame map summary:
[2021-09-14T04:23:03.152 BST DEBG ][ org.kde.kstars.ekos.scheduler] -   "/universe/Pictures/AllSky/Zenith/Light/Zenith_Light" : 385
[2021-09-14T04:23:03.153 BST INFO ][ org.kde.kstars.ekos.scheduler] - "Job 'Zenith' 60x60\"  completed its sequence of 0 light frames."
[2021-09-14T04:23:03.161 BST DEBG ][ org.kde.kstars.ekos.scheduler] - "Job 'Zenith' is configured to loop until Scheduler is stopped manually, has undefined imaging time."
[2021-09-14T04:23:03.161 BST DEBG ][ org.kde.kstars.ekos.scheduler] - Capture State "Capturing"
[2021-09-14T04:23:03.173 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:23:03: Driver indi_sx_ccd: sxClearPixels: libusb_control_transfer -> OK"
[2021-09-14T04:23:03.173 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:24:00.040 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:24:00: Driver indi_sx_ccd: sxClearPixels: libusb_control_transfer -> OK"
[2021-09-14T04:24:00.040 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:24:03.041 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:24:03: Driver indi_sx_ccd: sxLatchPixels: libusb_control_transfer -> OK"
[2021-09-14T04:24:03.041 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:24:03.792 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:24:03: Driver indi_sx_ccd: sxReadPixels: libusb_control_transfer -> OK"
[2021-09-14T04:24:03.793 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:24:03.943 BST DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 2900160
[2021-09-14T04:24:03.950 BST INFO ][           org.kde.kstars.indi] - "FITS" file saved to "/universe/Pictures/AllSky/Zenith/Light/Zenith_Light_476.fits"
[2021-09-14T04:24:03.993 BST INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 0.08 s, New Download Time Estimate: 0.08 s."
[2021-09-14T04:24:04.086 BST INFO ][   org.kde.kstars.ekos.capture] - "Received image 56 out of 60."
[2021-09-14T04:24:04.110 BST DEBG ][                       default] - WARNING: Phonon::createPath: Cannot connect  Phonon::MediaObject ( no objectName ) to  Phonon::AudioOutput ( no objectName ).
[2021-09-14T04:24:04.583 BST INFO ][ org.kde.kstars.ekos.scheduler] - Executing Job  "Zenith"
[2021-09-14T04:25:01.116 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:25:01: Driver indi_sx_ccd: sxClearPixels: libusb_control_transfer -> OK"
[2021-09-14T04:25:01.117 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:25:04.118 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:25:04: Driver indi_sx_ccd: sxLatchPixels: libusb_control_transfer -> OK"
[2021-09-14T04:25:04.118 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:25:04.867 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:25:04: Driver indi_sx_ccd: sxReadPixels: libusb_control_transfer -> OK"
[2021-09-14T04:25:04.868 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:25:05.020 BST DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 2900160
[2021-09-14T04:25:05.024 BST INFO ][           org.kde.kstars.indi] - "FITS" file saved to "/universe/Pictures/AllSky/Zenith/Light/Zenith_Light_476.fits"
[2021-09-14T04:25:05.066 BST INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 0.08 s, New Download Time Estimate: 0.08 s."
[2021-09-14T04:25:05.068 BST WARN ][                       default] - QProcessPrivate::createPipe: Cannot create pipe 0x6e95414: Too many open files
[2021-09-14T04:25:05.157 BST INFO ][   org.kde.kstars.ekos.capture] - "Received image 57 out of 60."
[2021-09-14T04:25:05.183 BST DEBG ][                       default] - WARNING: Phonon::createPath: Cannot connect  Phonon::MediaObject ( no objectName ) to  Phonon::AudioOutput ( no objectName ).
[2021-09-14T04:25:05.582 BST INFO ][ org.kde.kstars.ekos.scheduler] - Executing Job  "Zenith"
[2021-09-14T04:26:02.346 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:26:02: Driver indi_sx_ccd: sxClearPixels: libusb_control_transfer -> OK"
[2021-09-14T04:26:02.346 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:26:05.233 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:26:05: Driver indi_sx_ccd: sxLatchPixels: libusb_control_transfer -> OK"
[2021-09-14T04:26:05.233 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:26:05.982 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  "2021-09-14T03:26:05: Driver indi_sx_ccd: sxReadPixels: libusb_control_transfer -> OK"
[2021-09-14T04:26:05.982 BST DEBG ][           org.kde.kstars.indi] - INDI Server:  ""
[2021-09-14T04:26:06.146 BST DEBG ][           org.kde.kstars.indi] - Image received. Mode: "Normal" Size: 2900160
[2021-09-14T04:26:06.153 BST INFO ][           org.kde.kstars.indi] - "FITS" file saved to "/universe/Pictures/AllSky/Zenith/Light/Zenith_Light_476.fits"
[2021-09-14T04:26:06.197 BST INFO ][   org.kde.kstars.ekos.capture] - "Download Time: 0.09 s, New Download Time Estimate: 0.08 s."
[2021-09-14T04:26:06.200 BST WARN ][                       default] - QProcessPrivate::createPipe: Cannot create pipe 0x6e95414: Too many open files
[2021-09-14T04:26:06.287 BST INFO ][   org.kde.kstars.ekos.capture] - "Received image 58 out of 60."
[2021-09-14T04:26:06.298 BST DEBG ][ org.kde.kstars.ekos.scheduler] - Capture State "Image Received"
[2021-09-14T04:26:06.299 BST DEBG ][ org.kde.kstars.ekos.scheduler] - "Searching in path '/universe/Pictures/AllSky/Zenith/Light', files 'Zenith_Light*' for prefix 'Zenith_Light'..."

Do you see similar logging?  I did wonder if maybe there is some loose handling around counting the number of files in the target directory which causes the ulimit number of files to be exceeded.

And what is this?

default] - WARNING: Phonon::createPath: Cannot connect  Phonon::MediaObject ( no objectName ) to  Phonon::AudioOutput ( no objectName ).

I did set

ulimit -n 4096 (previous llimit was 1024) but checking just now with ulimit -a shows it has not taken the setting (or maybe I rebooted after setting).

astroberry@astroberry:~ $ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62253
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 95
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62253
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Jasem - can you check the code around this area to see if file handles might be being left open during the counting please?  Also maybe worth checking your ulimit values since you may have increased this - I know I have done this previously when coding.  Maybe this is why you could not reproduce the issue locally?

Without setting up a full C++ dev environment I don't know how much code investigation I can provide (I am a Java dev) and I know it is C/C++ code.  But if you can point me to the git source and the class(es) involved in the file counting I can see if I can understand it a but more?

I'll try and set a gdb session up for tonight's run and see if that helps.

Cheers!
Iain

I recently did a full upgrade on AstroBerry so will try this again as I've not had a chance to do much testing since my original post.

One thing to note from memory I had the error in the following two circumstances (1) one target with a lot of images all going into the same folder, and (2) multiple targets with different folders.

I am sure there are some out there asking why take hundreds of images of a target like this, but lucky imaging can be very good for splitting double stars.

Glad to see you contributing James.  It does seem that large numbers of files can lead to this issue, however that’s still a valid use case in my eyes.  I found another AllSky camera user who had reported the same thing.

Right now I’m looking for an easy way to obtain the ekosdebugger package/application which should help us to diagnose the issue.

I have cloned the source from git but seems there are many dependencies to apt-get before I can build it locally.

If anyone has the repository info where I can apt-get it directly then that would be appreciated.

Many thanks
Iain