Hi all,

I had a Kstars crash last night. Kstars was running on my Mac with Indi Server on my Rpi connected to my equipment. I was running an unattended scheduled sequence. It had been going for about 1 1/2 hours. It had completed the Ha filter and moved to OIII. It has done 3x300sec subs, completed the dither and settle, restarted guiding and was about a minute into capturing the 4th sub when it crashed. The analyze log (attached) shows this.

I wasn't logging to file at the time so no help there. I'll turn this on going forward; any ideas on topics to log would be appreciated.

The Mac crash report says:

terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
abort() called

I can't see much else in there to give a clue as to the cause (log attached).

The guide log ends with 

Guiding Ends at 2021-08-28 00:40:14
Log closed at 2021-08-28 00:40:14

I'm not sure if this is related to the cause (why would guiding end?) Hi all,

I had a Kstars crash last night. Kstars was running on my Mac with Indi Server on my Rpi connected to my equipment. I was running an unattended scheduled sequence. It had been going for about 1 1/2 hours. It had completed the Ha filter and moved to OIII. It has done 3x300sec subs, completed the dither and settle, restarted guiding and was about a minute into capturing the 4th sub when it crashed. The analyze log (attached) shows this.

I wasn't logging to file at the time so no help there. I'll turn this on going forward; any ideas on topics to log would be appreciated.

The Mac crash report says:

terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
abort() called

I can't see much else in there to give a clue as to the cause (log attached).

The guide log ends with 

Guiding Ends at 2021-08-28 00:40:14
Log closed at 2021-08-28 00:40:14

I'm not sure if this is related to the cause (why would guiding end?) or an effect of cleanup of the exception before Kstars went down?

Anyway, any suggestions would be much appreciated.

TIA

Regards,
John.

So I was out agin last night and I had another 2 Kstars crashes. The pattern is the same so I'm fairly confident its the same issue each time. I had full logging on (all boxes ticked). I've enclosed the log.

So the only thing logging at the time of the crash is the internal guider. I've worked out a little bit more of the sequence just before the crash.
- sub exposure completes and downloads
- guider starts dither and set state to dithering
- guider completes dither and set state back to guiding
- 13 secs later capture starts the next sub
- guider continues taking about 30 guide frames and working out guide pulses
- 22:07:50 guider takes another frame
- 22:08:21 guide frame times out. Says restarting exposure.
- 22:08:21 Kstars crashes with a SIGABRT

So I don't know why the guider doesn't receive the subframe... could be a comms error? Anyway, obviously that shouldn't cause a crash. I have seen this message before and guiding usually recovers.

After the two crashes I turned dithering off. This crash problem seems to have started at about the same time I started using dithering. After turning it off and restarting Kstars, the sequence completed without further error. So anecdotally it "might" be related to dithering.

Any ideas anyone?

Hi John, what about running guiding with PHD2 on your RPi?

The timeout is a good hint and needs deeper investigation.

Hi Wolfgang, thanks for the suggestion. I'll have a think about doing this. I was hoping to try Hy's new guider version which if I go to PHD2 I wouldn't be able to do, but obviously I do need something to stop these crashes as I've had multiple crashes for the last 3 sessions!

So, last night was clear again. Unfortunately I had 2 more crashes that followed the same pattern. The first happened whilst I was monitoring things; the second after I'd gone to bed. I had turned dithering off yesterday so that eliminates dithering from being part of the problem. The guider exposure timeout and restart message appears from time to time. Sometimes everything recovers, sometimes shortly after Kstars crashes. Everything seems to "freeze"... the guider freezes (graph stops - which makes sense) the countdown timer on the camera tab and on the main setup tab all stop (this makes less sense). This goes on for a few seconds and then either everything recovers and continues, or Kstars crashes.

That’s interesting. Could you please post the KStars log?

This is the log from the crash for my original post. I turned logging on for everything except the drivers. Am thinking now I should try again with the "CCD" and "Detector" driver logging turned on.

 

Seems like the crash happens when KStars is waiting for an input to be completed (the expected guiding frame) and in parallel sends an ABORT or a new capture request to the INDI service.

So it was clear last night and I tried again and had my system running from 9:00pm local until I had another Kstars crash (same details in the crash report) at 2021-09-05T02:26:15. This time I had INDI logging and CCD driver logging enabled. Log attached. So:
- it finishes with a sub, dithers, waits
- starts a new sub at 2021-09-05T02:25:43
- guiding is ongoing and a guide frame is uploaded at 2021-09-05T02:25:46
- then nothing for the next 30secs
- then...
[2021-09-05T02:26:15.392 BST INFO ][     org.kde.kstars.ekos.guide] - "Exposure timeout. Restarting exposure..."
[2021-09-05T02:26:15.395 BST DEBG ][           org.kde.kstars.indi] - INDI server disconnected. Exit code: -1

So no real new information. I have the indi server running on a rpi4 under Astroberry, so i poked around a little bit there. Looks like the indiserver.log file in /tmp is the most recent file (I'm guessing the existing file gets overwritten when indiserver is reSo it was clear last night and I tried again and had my system running from 9:00pm local until I had another Kstars crash (same details in the crash report) at 2021-09-05T02:26:15. This time I had INDI logging and CCD driver logging enabled. Log attached. So:
- it finishes with a sub, dithers, waits
- starts a new sub at 2021-09-05T02:25:43
- guiding is ongoing and a guide frame is uploaded at 2021-09-05T02:25:46
- then nothing for the next 30secs
- then...
[2021-09-05T02:26:15.392 BST INFO ][     org.kde.kstars.ekos.guide] - "Exposure timeout. Restarting exposure..."
[2021-09-05T02:26:15.395 BST DEBG ][           org.kde.kstars.indi] - INDI server disconnected. Exit code: -1

So no real new information. I have the indi server running on a rpi4 under Astroberry, so i poked around a little bit there. Looks like the indiserver.log file in /tmp is the most recent file (I'm guessing the existing file gets overwritten when indiserver is restarted?) so I don't think I have the indiserver.log file from last night.

So I'm thinking I'll try monitoring this log file next time and see if it gives anymore information.

The other thing I'm thinking about is increasing system resources to indilib. I see on the indilib menu there is a buffer size parameter set to 1024mb. I suspect that this would only be relevant to an internal indiserver within Ekos (but I could be wrong) and I suspect I would need to change the indiserver startup on the rpi under Astroberry. If anyone knows how to do this (or has any other views on the above) I would be very grateful to hear about it.

 

Did you manage to resolve this?  I have been having similar problems - Kstars crashes, the mount keeps tracking.  Same thing last night (on a v clear night!).  What should have been 4 hrs of sub returned 27 mins.  Ah well.  The only thing I did differently last night was that I left guide at Bin 1x1 (normally I leave it 2x2) - no idea whether that might be a factor, not a software person!

Cheers

Hi Vin,

I still have the problem, crashed overnight last night again (same error and symptoms). Yesterday I changed a couple of things but no improvement:
1. I saved the images to the Pi - usually save them on my laptop.
2. Set guiding exposure to 2 secs (usually 1) and binned 2x2 (usually 1x1).

I was trying to minimise Wifi traffic as I "suspect" it could be related to this (no proof though). There was a suggestion to use PHD2 guiding on the rpi - I currently use the internal guider on my Mac. I don't really want to do this and it will be the last resort (also no proof this will fix the problem).

Regards,
John.

 

At least I could reproduce it. If you start KStars with INDI locally, connect the devices (e.g. simulators) and then manually kill the indiserver process, KStars crashes - sometimes.

To reproduce it, set a breakpoint at ClientManager::serverDisconnected(int exit_code) line 275, start KStars with local simulators and then issue a killall indiserver. If you try to step through, you will see a SEGV violation after one or two steps.

Confirmed. Pushed a fix to INDI. Can you please recompile INDI from Git and then recompile KStars and test again?