×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

kstars/StellarSolver vs Canon Raw

  • Posts: 8
  • Thank you received: 0
Hi,
I'm running a kstars build from the most recent github source, so I could get the latest libraw with support for CR3 format and I've been bumping into a number of issues.
I'm connecting to a Canon M50 camera.  I can launch the profile and preview images in the FITS viewer.  However, capture only works in native transport format, not FITS. Not a big issue for regular capture, but it seems StellarSolver recently had a change that broke loading from native format, and so it fails to load the captured image from /tmp (it was loading fine a few days ago, before my most recent git pull- is there a preferred place to report these kinds of bugs?)

In addition to this issue, if I switch to my guide camera for plate solving, I successfully capture images, but kstars hits some memory limit and prints a stream of failed mmap calls to the console while loading the index files (as if no further memory can be allocated).  My process image is only about 1.1GB, running on a raspberry pi 4 with almost 8GB ram.  I'm not sure why this operation is failing or if I need to increase some OS limit to enable the process image to grow larger.

Feel free to point me to a better forum if this isn't the place to raise kstars bugs.
 
2 years 1 month ago #80079

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0
Here are some more details from stderr:

1. Canon driver configured with FITS transfer. Probably not an issue for regular light/dark capture, since native works fine, but this is a clue about what might be going wrong with StellarSolver capture- exactly the same error message appears (with a different tmp file).

org.kde.kstars.indi: Canon DSLR EOS M50 :  "[INFO] Starting 1 seconds exposure. "
org.kde.kstars.indi: Canon DSLR EOS M50 :  "[ERROR] Cannot open /tmp/indi_2DMHET: Unsupported file format or not RAW file "
org.kde.kstars.indi: Canon DSLR EOS M50 :  "[ERROR] Exposure failed to parse raw image. "


Switching to my ZWO guide camera and attempting to plate solve, I see dozens of failed mmap messages like this:
org.kde.kstars.ekos.align: "Capturing image..."
Found one coordinate representation.
org.kde.kstars.ekos.align: "Image received."
system: Cannot allocate memory
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/util/fitsbin.c:541:read_chunk: Couldn't mmap file "/home/astroberry/.local/share/kstars/astrometry/index-4204-04.fits"
system: Cannot allocate memory
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/util/fitsbin.c:541:read_chunk: Couldn't mmap file "/home/astroberry/.local/share/kstars/astrometry/index-4204-03.fits"
system: Cannot allocate memory
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/util/fitsbin.c:541:read_chunk: Couldn't mmap file "/home/astroberry/.local/share/kstars/astrometry/index-4204-03.fits"
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/util/quadfile.c:135:my_open: Failed to open quads file
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/util/index.c:417:index_reload: Failed to read quads from /home/astroberry/.local/share/kstars/astrometry/index-4204-03.fits
/home/astroberry/Projects/stellarsolver/stellarsolver/astrometry/blind/engine.c:204:engine_add_index: Failed to load index from path /home/astroberry/.local/share/kstars/astrometry/index-4204-03.fits
system: Cannot allocate memory
2 years 1 month ago #80092

Please Log in or Create an account to join the conversation.

  • Posts: 2876
  • Thank you received: 809
Hi there,

I am currently working on some major revisions in stellarsolver, but they shouldn't affect the things you are trying to do here.  This message:

org.kde.kstars.indi: Canon DSLR EOS M50 :  "[ERROR] Exposure failed to parse raw image. "

Is not an error in stellarsolver.  If you look at the beginning, it says it is an indi error and if you search here: github.com/indilib/indi-3rdparty/search?...d+to+parse+raw+image 
you can see that it can only be one of two drivers,  indi-pentax/pktriggercord_ccd.cpp  or  indi-gphoto/gphoto_ccd.cpp .  Perhaps there was a recent change in one of those drivers?  Maybe there is an issue in your installation of gphoto?  

StellarSolver does not capture any images, it simply takes images and image buffers that have been given to it by KStars and plate solves them.  Are you using the internal StellarSolver solver to solve the images, or are you using ASTAP, online astrometry, or local astrometry.net?  StellarSolver has the ability to plate solve using different methods and I want to make sure of which method you are using to know what is happening.  It also matters which method you are using because the internal solver needs to receive an image buffer from KStars to solve the image which means KStars needs to be able to load the image (the driver error we discussed already could cause issues with that).  The other solving methods usually require FITS in order to solve the images, since astrometry.net itself cannot solve a RAW file.  If the image is loaded in KStars, StellarSolver can take the loaded image buffer from KStars and export a FITS file for those other solvers to work, so even if you use a Canon camera with KStars and are capturing RAW files, we can still plate solve using any method as long as the image loads in KStars.

Finally the mmap errors you reported at the end, they are related to your index files, not the file you are trying to solve.  Looking at what the output says, I am guessing that you have a LOT of index files?  That actually can take up quite a bit of memory, especially if it tries to load them all at once.  And note that you might have 8 GB of ram, but is your system armhf or arm64?  If it is not 64 bit, it can't use all 8 GB of ram, I think it can use 4 GB.  Your operating system will already take a little RAM and if you have a few GB of index files, you can quickly hit the limit if you load them together.  Now StellarSolver and astrometry.net have a setting called "in parallel" that lets you load index files sequentially or all at once.  There is a huge performance gain to turning on that setting, but if you don't have enough ram and you have too many index files, you can't do that.  To change that setting in KStars, you can edit your options profile you like to use most and turn off that setting.  Then see if it works better.  I wrote code that will automatically check to see if you have enough RAM on your system to load your index files and it is supposed to turn it off if you do not, but if you are on a system where you can't actually use all your available RAM, then that might give a false positive.

Also if you are having problems with recent changes I made in stellarsolver, you can go back to version 1.9 until I finish with my changes.  That is available for download from the stellarsolver repo.  I don't think I broke anything, but there were a lot of changes and it needs more testing.

Thanks,

Rob
Last edit: 2 years 1 month ago by Rob Lancaster.
2 years 1 month ago #80122

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0
Thanks for the detailed response.
As I understand it, the "Canon DSLR" driver is just a wrapper for the gphoto driver, and judging from the code you pointed to, even though I have configured the setting to capture native frames, when StellarSolver invokes the capture it's taking the FITS pathway (the /tmp file is only generated when FITS is selected for transport). I'm using the internal solver, so in principle it should just be receiving the raw image buffer. I will see if I can dig more to figure out why the wrong setting is used by the driver.

My device is armv7l which is 32 bit. I don't think I'm getting close to 4GB of allocated RAM, but I'll disable the parallel load and see if that fixes the issue. I'm a bit confused, though, about why a 32 bit pi would ship with 8GB of RAM on board- does the limit only apply to the address space of individual processes?

Thanks again, will follow up shortly.
2 years 1 month ago #80123

Please Log in or Create an account to join the conversation.

  • Posts: 2876
  • Thank you received: 809
I wouldn't really say the INDI GPhoto driver is a "wrapper" for the gphoto driver.  It is a program built on the foundation of libgphoto2, just like all the other ASCOM and INDI drivers are built upon the SDK's, dynamic libraries, and other programs provided by the camera manufacturers and other programmers.  An INDI driver for a camera is primarily a program that handles settings, functions, and communications with the underlying camera library to perform the tasks asked of it by KStars and other programs.  Many of the ones for telescope mounts have to handle all kinds of serial communications with the mount and keep track of what it is doing.  Then the INDI driver has to serve all that information up over the network for the client program.  If you look at the INDI driver it is actually pretty complex and involved.

For the FITS pathway, you have one misunderstanding.  It doesn't matter if you have the Canon camera taking FITS or native images, if you are using the internal solver, Kstars must load the image and then sends the image buffer directly to StellarSolver.  There should be no FITS file required unless you are using one of the external solvers.

To answer your question about the Pi, it is not a problem with the PI itself.  The Raspberry PI is a 64 bit machine and can run a 64 bit operating system.  My 8GB Pi is running a 64 bit system right now.  If you are using a 32 bit architecture it cannot create files nor can it give memory to any one program greater than ~3-4GB.  That is due to using 32 bit integers to represent the pointers in the address space. (I think I got that right).  I do think it is capable of using the other RAM, but not for the same process as you said.

thanks,

Rob

 
2 years 1 month ago #80124

Please Log in or Create an account to join the conversation.

  • Posts: 2876
  • Thank you received: 809
Can you share an image that you would like to have KStars open and then be plate solved by stellarsolver?  I can check to see if there are any issues.
2 years 1 month ago #80125

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0
I attached gdb to kstars and set a breakpoint at the failed mmap error message- when it fails, the process size has reached 3.1GB, which sounds close to the limit, (4GB - 1GB reserved address space, according to this post: stackoverflow.com/questions/37409250/why...bit-armv7l-succeeded). I verified that the option to load all files at once is unchecked. I'll trim my indexes down to the minimal set that I need and hopefully I won't hit this limit.

I'm still digging on the FITS issue.
2 years 1 month ago #80126

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0

Thanks, yes I understand. Just to clarify the situation: in the INDI control panel the driver is configured to use "native" transport, and when I capture a preview image in the Ekos CCD tab it successfully downloads an image and loads it in the FITS previewer. However, when I switch over to the plate solving tab and click capture and solve, for some reason the driver is ignoring the native setting and attempting to use FITS transport.  I can generate the same failure to open /tmp/indi_*** error if I switch the transport to FITS in the control panel and then try to capture a preview in the CCD tab.  I'm not sure why the setting is missed, will try to debug this flow.
Yes, happy to share once I verify I can get an image. I do see that if I switch to my ZWO camera and just snap some images indoors then the solver is running (but triggering the mmap failures).  If I take the scope outside and snap some frames, do you expect that the solver will run ok even if some index files fail to load?


 
Last edit: 2 years 1 month ago by Robert Sim. Reason: formatting
2 years 1 month ago #80153

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0
Hmm.. apparently quote doesn't work?
Last edit: 2 years 1 month ago by Robert Sim. Reason: formatting
2 years 1 month ago #80154

Please Log in or Create an account to join the conversation.

  • Posts: 2876
  • Thank you received: 809
Yes, even if there is maybe some sort of issue with an index file such as corruption etc, the solver should still be able to use the others to plate solve the image and it can often be successful.
2 years 1 month ago #80155

Please Log in or Create an account to join the conversation.

  • Posts: 8
  • Thank you received: 0
Returning to this topic after some time just reverting to my windows-based workflow. I'm really interested to get the PI-based workflow running. Before I set it down I did figure out some details about RAW capture and what settings I should apply.
There are still a couple issues I need to understand better:
- presumably most folks can run image capture, calibration, etc just fine in kstars with the 32bit astroberry server, and I wonder why in my case I'm running out of memory. Should I just pare down my index files to the bare minimum? Are there other things I should disable in kstars to keep the memory usage down? Alternatively, should I just proxy everything through the PI and run kstars on a 64 bit laptop? Any pointers on how to do this?
- should I accept that I need more than 4GB address space and try to find/create a 64 bit build of astroberry server? You mention that you're running a 64bit OS so presumably you've had some success finding an image and getting it set up- are there any pointers available to get started?
1 year 8 months ago #84007

Please Log in or Create an account to join the conversation.

  • Posts: 155
  • Thank you received: 12
Hi

not sure I’m going to help here but I run kstars/ekos on a 4gb Pi using Astroberry so currently 32 bit. I have a sony a7rii so using a gphoto related indi driver. Camera takes raw but transports it in fits format. I’ve got a large chunk of the index files installed and when stellarsolve runs it appears to check and then says it cannot load in parallel but continues and solves pretty quickly. So unless there’s some weirdness around having 8gb I would have expected it to work.
Have you tried using the astap or watney based solvers to see if they work? They have their own index files and internal star finding routines.

I seem to remember reading somewhere that it’s pretty simple to set the linux kernel to be 64bit even without having to have all the programs converted over. Didn’t look to be any help for me given i’ve only got 4gb but might be worth investigating but then you should be working fine anyway…


Regards

Nigel
1 year 8 months ago #84026

Please Log in or Create an account to join the conversation.

Time to create page: 1.145 seconds