×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

EKOS imaging pipeline?

  • Posts: 33
  • Thank you received: 5
I'm looking at setting up a pipeline for Speckle Interferometry. In a nutshell, I would like to pre-process images as the frames are taken. Conceptually this would be from a remote site. We take as many as 10,000 frames per target and while small, that's a lot of data.

Not sure how to even search for what I am asking for, but want the save to disk function in the observatory and the scheduling to be done in my office. That is EKOS scheduler is running on my system in the office, but the data is being collected and reduced at the observatory. Once processed the result is about the size of a single frame. This is of interest to anyone doing high cadence imaging.

Can this be done now? If so a hint on how would be nice. Also a pointer to where you would recommend hacking in the cuda code for the processing :-)

Thank you,
Last edit: 5 years 11 months ago by Greg Jones.
5 years 11 months ago #24749

Please Log in or Create an account to join the conversation.

Replied by Jasem Mutlaq on topic EKOS imaging pipeline?

Greg, this is quite interesting. It is possible, but might require more work to get done. In the Capture module, there is a "script" field which you can execute after each image is captured so it can then process the image. Currently, it does not pass any argument nor it sets any environment variables about the last captured frame, but this could be added in. Is this what you're looking for?
5 years 11 months ago #24756

Please Log in or Create an account to join the conversation.

  • Posts: 33
  • Thank you received: 5

Replied by Greg Jones on topic EKOS imaging pipeline?

knro, I can make that work in theory, but it dosn't do what I am looking for.

Think remote observatory with slow data connection. We shoot about 5000 frames per target and 10+ targets an hour. We can fill a terabyte drive in 3 nights. So the goal is to collect the raw frames in the observatory and do the Speckle reduction (Fourier based) on the fly (GPU) returning just the Speckle results. That will give us close to a 5000 to 1 reduction in data needing to be transferred. We also need to put the files into a reasonable directory structure and Ideally store each target in a FITs cube instead of a frame per file.

I know that's a lot and I'll take a shot at it. Since the first note I've looked at the code and have an idea what to do.

TIt looks like EKOS does the actual file save, correct? That means that EKOS would need to be running on the observatory computer. If all that is correct, then splitting the scheduler from the capture module would be the Ideal way forward. That is, let the capture module run under the server, and direct it with the scheduler via the server.

Hope that makes sense?

I'm hoping to get the cuda/GPU portion working, probably with Open Astro, but CCDCiel is also a possibility as it uses INDI for the cameras from what I can tell.

Thank you,
5 years 11 months ago #24758

Please Log in or Create an account to join the conversation.

Replied by Jasem Mutlaq on topic EKOS imaging pipeline?

INDI driver can save the files at the observatory side. The scheduler can either be run remotely or on site. I think you need to write your own script/software to monitor the directory where INDI dumps the data and then perform your own processing there.
5 years 11 months ago #24759

Please Log in or Create an account to join the conversation.

  • Posts: 33
  • Thank you received: 5

Replied by Greg Jones on topic EKOS imaging pipeline?

Jasem,

That may be the best option. In fact the more I think about it, running the scheduler on the observatory hardware seems like a more robust solution (WAN disconnect). Nice thing about INDI is there are a number of options here. The local file processor was the initial plan, but for several reasons, doing it as part of the capture process looks like a better option.

I have some homework to do... Thank you again.
5 years 11 months ago #24761

Please Log in or Create an account to join the conversation.

Time to create page: 0.519 seconds