×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

Perfecting the Scheduler

Have you looked at CCACHE?
5 years 11 months ago #25389

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

I am actually using ccache in the project configuration. But it's not resulting in the acceleration I'm used to. Apart from the configuration issue where enabling the option calls "ccache /usr/lib/ccache/g++" which is a weird though perhaps harmless process duplication, cache hits are really low. But even at the dependency enumeration, cmake sometimes rebuild files I see no reason for it to. Probably my configuration...

-Eric
5 years 11 months ago #25391

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

As if slow builds weren't enough, I suffered a VM corruption this evening :) but my changes are being tested, scheduler is sleeping until twilight before capturing M51.

-Eric
5 years 11 months ago #25415

Please Log in or Create an account to join the conversation.

  • Posts: 1185
  • Thank you received: 370
Hi folks,
I am not sure whether I got all points right in the thread, so please excuse, if I bring up a topic here that is already addressed.

But first of all: the scheduler is THE killer application for me. Thanks a lot, it brings kstars on a completely new level.

Back to my problem. As far as I understood, when a job fails after several retries (for example during focussing), the job is marked as "ABORTED". But this might be too strict for example if clouds pass by.

So it could happen, that a certain job could not be executed for a while. If other jobs are present that could run, OK, let them run. But what happens, if all jobs are aborted? In my case, I only have one single job. In this case, EKOS is terminated, right? At least this happens in my case.

What would be the suggested behavior: I would like having a "retry after a certain amount of time" and restart the aborted job.

Does that make sense to you?
Cheers
Wolfgang
5 years 11 months ago #25631

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

@sterne-jaeger this absolutely makes sense, and that's what I work towards to. Ideally (my view at least), the scheduler should be able to plan ahead as many observations as possible, and be robust to as many error situations as possible. But we can't go too fast with features.

The recent changes were targeting scheduler jobs and sequence jobs, so that captures were counted properly. There are still issues, for instance when there is no filter wheel, and side effects, for instance when duplicated jobs all complete at the same time, but that will be solved soon.

The other activity was targeting job evaluation, so that jobs are aborted when the scheduler stops and re-evaluated when the scheduler restarts. The idea is to adjust the algorithm to always restart aborted jobs while the scheduler runs, either with a limited number of retries or with a lower priority. Here too, there are side effects to the preliminary fixes, for instance scheduled jobs are now assigned a fixed start time, but there is no provision yet to start such scheduled jobs in advance. This will be solved soon too.

-Eric
The following user(s) said Thank You: Jose Corazon
5 years 11 months ago #25635

Please Log in or Create an account to join the conversation.

  • Posts: 1119
  • Thank you received: 182
Thanks, Eric! The scheduler already works really well as is, thanks to your and Jasem's efforts.

The priorities as I see them are:
#1 and of foremost importance by far: Ensure that no imaging time is wasted during a clear night!

That already can be achieved with the scheduler as is, as long as a series of backup jobs are scheduled. That, together with the ability to reevaluate virtually assures that every minute of darkness is used for photon collection, even though the targets may not be worked off in order of priority. During a clear night, the scheduler works very predictably, at least in my hands.
The other issues can be solved by scheduling the same target repeatedly with different priorities. That can take care of the passing cloud contingency.

At any rate, the scheduler is vastly improved over what it could do only a few weeks ago. By all means, take your time with adding the remaining tweaks.

And yes, the scheduler is THE killer app!

Jo
5 years 11 months ago #25638

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

I'm testing my recent additional changes on the scheduler, and I observe that when using the filter wheel embedded in the CCD simulator, my sequence files requesting L, R, G, and B never process L.

I can see the capture tab requesting change to Red, Green and Blue, but when it is time to set the Luminosity filter, (1) there is no log in the capture tab, (2) the CCD states it is moving to filter 7, which is LPR, and (3) the capture is stored in folder "Blue".

This issue is causing a side effect to the scheduler, which cannot consider the job complete when there is a missing Luminosity set. BUT there seems to be an additional issue to counterbalance this, as the capture module notifies completion to the scheduler, which can thus continue. As a consequence, when starting the scheduler anew, all sequence jobs are considered uncompleted and restart.

This issue didn't arise when using my physical setup in the basement. However, there was another problem with the path to which the capture was stored, which did not contain a filter name when no filter driver was connected. I worked around this with a filter wheel simulator (not to be confused with the CCD simulator embedding a filter wheel above), but I'll have to come back to this later on.

-Eric
5 years 10 months ago #25728

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

I think I found the reason : the CCD simulator has a "Luminance" filter, and the filter wheel simulator has a "Luminosity" filter. Hence sequence files are somehow incompatible between my physical setup and my simulated setup.

-Eric
5 years 10 months ago #25730

Please Log in or Create an account to join the conversation.

I'll update Luminosity --> Luminance in the filter wheel simulator as well.

EDIT: Actually, they should be the same. You probably have this saved as config from a long time ago.
The following user(s) said Thank You: Eric
Last edit: 5 years 10 months ago by Jasem Mutlaq.
5 years 10 months ago #25734

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

I'll have to check: I created the sequence while connected to my physical setup, before I realized there was no filter wheel in said setup. Thus I think the unexpected filter name is not sourcing from the INDI setup, but from the capture tab itself. But that's maybe what you mean.

-Eric
5 years 10 months ago #25735

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Perfecting the Scheduler

@knro any pointers on how I can write tests for the scheduler? I'm preparing several scheduler/sequence files and I'd like to automate their execution and verification.

-Eric
5 years 10 months ago #25805

Please Log in or Create an account to join the conversation.

  • Posts: 90
  • Thank you received: 37
@TallFurryMan: Writing tests for the KStars GUI is difficult, I recommend to forget about it for the scheduler.
The following user(s) said Thank You: Eric
5 years 10 months ago #25809

Please Log in or Create an account to join the conversation.

Time to create page: 0.290 seconds