I have a ZWO ASI290MM CMOS camera which I use for acquiring large numbers of short exposures (big mirror, bad mount and lucky imaging!). My exposure times are typically 500ms. I have been using Ekos/INDI on Linux over the past 8 months, up-to-date with the Fedora 27 Bleeding channel. I enjoy immensely the KStars/Ekos/INDI combination, however my observations are affected by a number of issues when working with very short exposures (sorry if this has been answered elsewhere, I could not find anything about it):
- Whatever the Ekos task (sequence queue, focusing), there seems to be an incompressible delay of ~0.5s between exposures on my setup, making me loose about half of the observing time currently! This delay remains the same when running the CCD simulator, even with tiny frame sizes, so it does not seem to be related to the ASI hardware/driver. I have a fast PC equipped with SSDs hence hardware should not be the limitation here. Actually, I measure the same delay when acquiring images from a remote PC (through GbE), so it really feels like some kind of software scheduling limitation. Is there a way to reduce this delay?
- Although I thought
it should not be the case since INDI V1.3
, I still cannot enter a number of images >999 in the "Count" widget of the Sequence Queue dialog. Am I missing something?
- When running large sequences cumulating more than 1000-2000 exposures I almost systematically get into the issue where the Ekos sequence queue task aborts saying that it cannot write the image, although there is still plenty of disk space (this last issue I understand would need to investigate more in order to provide you with a complete log, please tell me how I should do it).
Thanks, and kudos to the developers for this amazing software!
I'm at work right now so I can try test with simulators. There is always some overhead, but there are ways to reduce that.
1. Disable FITSViewer in FITS Options.
2. Use "Local" mode so that the images are not transferred back to client and are instead directly saved to the disk.
3. Optimize drivers
For #3, I found that I could really speed things up by doing a couple of changes in the drivers:
1. Start the next image immediately when the prior one complete before even saving to disk.
2. Increase polling frequency.
I'm actually working now on adding a looping mode. Will let you know of the results.
Great! Thank you for your quick reply Jasem!
- Yes I already tried disabling the FITS viewer and save locally, but it does not do much, so I am eagerly looking forward to testing your solution #3!
While I do not do many exposures under 1 second, I am interested in the ASI indi driver, and so I added some timers to the driver and ran several tests using an ASI 174mini (USB2) and an ASI294mc (USB3).
What I found surprised me.
With an exposure of 0.000032 seconds the driver spent on average over 300,000 usecs (0.3 second) polling ASIGetExpStatus. It only only spent around 18,000 usecs (18 msecs) downloading the image via USB2 at 40% bandwidth.
I tried again with an exposure of 0.5 and got the same overhead.
When the remaining exposure time drops to (or starts) below 1.1 seconds the indi ASI driver calls ASIGetExpStatus every 10,000 usecs (10 msecs), and for the shortest exposures this function is called over 30 times each, and for a half second exposures it is called over 85 times each.
I then repeated the tests with an ASI294MC in 8 bit mode and the image size reduced to the same as that of the 174 and saw the same overhead.
This implies to me that there is a consistent delay or overhead in either the ASI provided SDK or the ASI camera firmware.
I recommend starting a thread on the ASI forum -- in fact I will do so tomorrow (after searching if any related threads).
Thanks Jasem, your changes seem to have been effective!
I just ran tests with kstars-bleeding-2.9.2-7.2 and libindi-bleeding 1.6.2-10.2 on my Fedora 27 system:
- I can now generate sequences with thousands of exposures
- The delay between exposures with my ASI290MM has been reduced to 300ms (was 500ms), which is inline with wulfblat's measurements. It does not depend much on the acquired frame size. Now and then the acquisition seem to pause for 1 or 2s (the countdown is restarted without exposures being acquired), but no big deal.
- With the simulator and tiny frames the delay is now about 100ms and very steady.
- It would be great if one would be able to specify a Delay < 1s in the sequence queue dialog (for some reason I am able to insert a comma in the widget, but not a decimal point, although my Desktop is localized with the English notation).
Now the bugs:
- Thanks to the increased throughput I was able to reproduce the annoying bug I was mentioning in my previous post, using short exposure times. So far it has only shown up while in the "Upload: Client" mode, even though I am not acquiring from a remote computer. From time to time, Ekos decides to abort a sequence after a few hundred to a few thousand exposures with the following type of error message:
2018-02-20T11:42:57 Failed to save file to /acquisition/test/Light/Red
I checked that there was still plenty of disk space left in all partitions. The bug happens with the CCD simulator too, so it is pretty easy to reproduce in a matter of 10-30min, e.g., by requesting a sequence of 10000 0.01s exposures.
- Another issue which may be more related to ASI driver is the fact that after completing (or aborting) a large sequence I often experience the following problems:
-- The KStars and Ekos windows freeze after pressing the Device "Disconnect" button, while consuming 100% CPU. The only thing I can do (after waiting a few minutes) is to kill KStars. When restarting KStars, Ekos and the INDI server, I get the message that the INDI server is already running, which is not an issue, as I am able to restart it.
- When I manage to disconnect and stop INDI, while trying to restart it, I get the following error message: "INDI server failed to start: Too many open files.", in which case the only thing it seems I can do is to restart KStars.
Thanks again for your help and sorry for the long post!
No I did not. I just tried it: when it is activated, in the sequence queue one single exposure is completed and then nothing more happens for both ASI and the CCD simulator. The spinning widget still turns, and I can abort the sequence, but that's it. The first time I tried it with the simulator the whole KStars crashed immediately when I started the sequence.
To try out the feature, you'd either have to build KStars from GIT, or wait until the next release. But it looks like this feature will be limited anyway. So the premise is that as SOON as the capture is done, it immediately starts the next capture and THEN uploads/saves the image. So this would be useful IF the capture time is longer than the transfer time. So if each image is 10 seconds long, but the transfer takes 2-3 seconds, then fine. But if the capture time is 0.5 seconds, but it takes 1 second to upload the image to the client, then this would fail miserably.
But even if you're taking 5 minutes captures (like what I usually do), it can help. In my case, it takes 20 seconds to upload the image back to KStars. So with the regular method, after an image is capture AND uploaded, the next image capture begins, but with rapid looping turned on, then it begins immediately. Therefore, you can save 1 minute of time every 15 minutes of capture.
OK thanks, I will try with the GIT version! Transfer time with a 4MB image is significantly less than 0.5s when writing directly to an SSD, even using CFITSIO . Well anyway for now the limiting factor seems to be the ASIGetExpStatus() issue.
btw I hope you will be able to reproduce the "Failed to save file to" bug with large sequences in the "Upload: Client" mode. In practice this is the most annoying issue with high acquisition rates.