×

INDI Library v1.7.8 Released (11 May 2019)

Mid year release of INDI Library v1.7.8 comes with a few drivers and improvements to to existing drivers.

Full Automation and Weather handling in the Scheduler

  • Posts: 44
  • Thank you received: 19

Full Automation and Weather handling in the Scheduler was created by H__

Hi all. I want to use EKOS in a fully automated observatory setup.

The scheduler has most basic parts that are needed except complete weather service handling.
The first part exists; wait with starting an imaging run until the weather is OK.
However the second part, aborting an imaging run, parking the mount and closing down the roof or dome of the observatory in case of bad weather conditions is not implemented.
This second part is crucial to me.

A start has clearly been made in scheduler.cpp but the weather handling code is all commented-out and also incomplete.
I need this work completed and while I can work on this myself I prefer to do so in cooperation with the original author(s). Maybe there are already plans or even code on a branch somewhere to finish this ?

I'm new to KStars code but not to INDI code. Who can help me explain the weather handling design in KStars, how QDBus and INDI are combined, why checkWeather() does not exist but isWeatherOK() does etc. What the reasons were the code was commented-out instead of completed, etc. And advice on how to proceed.

-- Hans
The following user(s) said Thank You: knro, fenriques
1 month 5 days ago #38029

Please Log in or Create an account to join the conversation.

Replied by knro on topic Full Automation and Weather handling in the Scheduler

All great questions Hans!

1. DBus: Only the scheduler uses DBus to communicate with the Ekos. At face value, it doesn't make sense to use DBus within the same process, but the scheduler was planned from the beginning to be its own separate niche. Using the same logic in the scheduler, anyone can develop an independent scheduler using the same dbus calls.
2. There was _some_ preliminary support for weather ALERT causing a shutdown of the observatory, but as with most things, it's a lot more complicated than that. Specific actions need to be taken when such a condition exists. What about WARN? maybe the observatory can temporarily close the roof..etc and then reopen again? All this logic is currently not handled at all. So it was a choice between implemented a borked weather handling implementation or none at all, so what you see in the code is the relic of the initial attempt at this.

I think to proceed, you probably need the following:

1. A dialog to set what actions are required when a specific weather condition state changes

Weather_Param
State
Action

Weather Params are already defined and standard. States are OK, BUSY (WARN), and ALERT

ACTION could be like Shutdown, Resume, Email, Ignore, Script (Run some script), Suspend (i.e. stop capture)

Then you can have a checkbox [ X ] Auto Resume, that is , if the weather condition improves, should the observatory resume operations or remain at shutdown?

So these are my initial thoughts on this.
Jasem Mutlaq
Support INDI & Ekos; Get StellarMate Astrophotography Gadget.
How to Submit Logs when you have problems?
Add your observatory info
The following user(s) said Thank You: El Corazon
1 month 4 days ago #38039

Please Log in or Create an account to join the conversation.

  • Posts: 111
  • Thank you received: 23

Replied by fenriques on topic Full Automation and Weather handling in the Scheduler

Hans,
I'm using my observatory fully automated from remote as well and thus relying completely on the scheduler; having weather condition controlling the shutdown of the observatory would be a great improvement (and relief). Thank you for taking care of it.
I could provide some help defining and testing use cases if needed.

Ferrante
3 weeks 13 hours ago #38706

Please Log in or Create an account to join the conversation.

  • Posts: 158
  • Karma: 2
  • Thank you received: 40

Replied by sterne-jaeger on topic Full Automation and Weather handling in the Scheduler

Hi Hans,
that sounds like an excellent idea! Just one thing: please try to implement weather handling standalone and do not try to squeeze it into the scheduler. We plan to rework the scheduler currently, since it is a huge and awfully complex class.

It should be easy to change the scheduler in such a way that it can be started/stopped/aborted/paused/... from outside. That could be a way to let the two modules easily cooperate without deeply intertwining them.

And please try to consider automated tests from the very beginning.

If you need help, do not hesitate...

- Wolfgang
The following user(s) said Thank You: knro, El Corazon
3 weeks 8 hours ago #38719

Please Log in or Create an account to join the conversation.

  • Posts: 325
  • Thank you received: 50

Replied by Ron Kramer on topic Full Automation and Weather handling in the Scheduler

I haven't used the scheduler much yet - but this is a necessity for me as well. I THOUGHT the weather watcher was already capable of doing this? IT won't allow me to open if cloudy, but you're saying it won't shut down and park/close?

I have a script that I run from windows. Is there something similar that I can run from linux?
The cloud watcher flags unsafe/clouds or whatever to its data file.
Is there a way to read that file during sub acquisition and abort or wait until (between exposures) to check that file and run a shut down/close up script?

I use to leave SGP running and go to bed - no worries. It would always be closed in the morning because even daylight is unsafe. So it would close at dawn.

So the dat file is written every minute with this format - Really it could look only at SAFE or UNSAFE and close up if unsafe. The rest could be ignored
since unsafe is flagged in the event of any of the other values aren't in the GREEN zone.

"2019-05-03","18:07:21","Cloudy","Dry","Very light","1.2","15.9","4014","1","15.9","10%","21.0","","Closed","5","0","Unsafe","Calm","3.0","Humid","77%"

Maybe you're not referring to Weather Watcher driver? What about watch dog? Can it watch the dat file from the cloud watcher and close up?

This was my windows .vbs script file.
dim dome
set dome = CreateObject( "POTH.dome" )
call dome.CloseShutter()
set dome = nothing

The cloudwatcher software would call the script. But I can't run the cloudwatcher software under linux.
I could run the cloudwatcher software on the desktop running Kstars in the house, but it would have to send a command to the nexdome driver to close.
AP Mach1 / CP4 APCC & PEMpro.
EXP SCI - ED152cf APO - Celestron 11" RASA - Stellarvue 80mm
Baader F2 HS NB filters, Lodestar X2 guide camera / OAG - ZWO 290mm mini
ZWO ASI1600MM Pro / ASI174M (solar) / ASI094MC
NEXDome, CLoudwatcher, AVX mount/ASIair and Stellarmate

Last edit: 2 weeks 6 days ago by Ron Kramer.
2 weeks 6 days ago #38751

Please Log in or Create an account to join the conversation.

  • Posts: 44
  • Thank you received: 19

Replied by H__ on topic Full Automation and Weather handling in the Scheduler

Hi Ferrante,

fenriques wrote: I could provide some help defining and testing use cases if needed.


To get this started without having to change anything in EKOS I've written ekos_sentinel.py. It basically adds what is missing today :
weather safe ?
yes:
    scheduler running ?
    yes:
        ok
    no:
        start scheduler
no:
    scheduler running ?
    yes:
        stop scheduler
        park mount
        close roof
        warm ccd
    no:
        ok

When EKOS gets bad-weather-shutdown handling itself this external daemon can still prove useful as it can be made into a safeguard for when EKOS crashes or locks up. So I intend to maintain it.
Code lives at https://github.com/d33psky/rolloffroof/tree/master/ekos

Here's my 3th test run log (I went to sleep around 03:00 :) ) :
./ekos_sentinel.py --indi_host localhost
2019-05-01 02:45:51,250 ekos_sentinel INFO     weather is safe, roof is closed, start ekos scheduler
Start scheduler
2019-05-01 02:46:51,306 ekos_sentinel INFO     weather is safe, roof is open
...
2019-05-01 03:32:53,546 ekos_sentinel INFO     weather is safe, roof is open
2019-05-01 03:33:53,565 ekos_sentinel WARNING  weather is unsafe, roof is open, stop ekos scheduler
Stop scheduler
2019-05-01 03:33:53,567 ekos_sentinel WARNING  park mount
2019-05-01 03:33:54,583 ekos_sentinel WARNING  close roof
2019-05-01 03:34:51,630 ekos_sentinel WARNING  warm camera
2019-05-01 03:34:51,651 ekos_sentinel WARNING  failed to warm the camera, this is not critical to safety so just continue
2019-05-01 03:35:51,725 ekos_sentinel INFO     weather is unsafe, roof is closed

It did the critical things right; stop the EKOS scheduler, park the mount, park the roof, but it failed to warm the camera. I'll work on that of course ;-)
I also need to add cap support, and allow for a flexible shutdown order as that is really observatory dependent. I also should replace calling indi_getprop and indi_setprop with the Python pyindi-client, which I learned existed only today.

The interfacing with EKOS goes via DBUS and there's things in EKOS that can be improved as well like query capabilities (assuming DBUS even allows for that).

-- Hans
The following user(s) said Thank You: Gonzothegreat, fenriques
2 weeks 6 days ago #38752

Please Log in or Create an account to join the conversation.

  • Posts: 111
  • Thank you received: 23

Replied by fenriques on topic Full Automation and Weather handling in the Scheduler

thanks Hans,
Seems perfect. I will give it a try. Btw I see that we share part of the equipment (10Micron, ASI).
A few questions:
- the workflow manages the scheduler to start if the weather is ok. But also resumes a job if after a break the weather returns ok? it seems it doesn't.
- I guess it should be run with a command like:
python3 ./ekos_sentinel.py 192.168.xxx.xxx --options
Can you provide a simple start command line please?
- For my setup I'll add some time delays (sleep), for example my roof hits the mount if it is not yet parked (I have parking sensors, but better be safe). It would be nice to have also a fixed number of retries, e.g. try to warm up the camera for 3 times before issuing a warning.

Ferrante
2 weeks 6 days ago #38758

Please Log in or Create an account to join the conversation.

  • Posts: 44
  • Thank you received: 19

Replied by H__ on topic Full Automation and Weather handling in the Scheduler

Hi Wolfgang,

sterne-jaeger wrote: please try to implement weather handling standalone and do not try to squeeze it into the scheduler. We plan to rework the scheduler currently, since it is a huge and awfully complex class.

The scheduler class would certainly benefit from refactoring. Is your rework started somewhere yet ? Branch/posts/chat ?

It should be easy to change the scheduler in such a way that it can be started/stopped/aborted/paused/... from outside. That could be a way to let the two modules easily cooperate without deeply intertwining them.

Agreed. As a quick hack I've already used what's available via DBUS today from a python script. Should such a weather or safety (broader) tab use DBUS to instruct the scheduler or should we use plain C++ class methods ?

And please try to consider automated tests from the very beginning.
If you need help, do not hesitate...

I will definitely need help. I'll first continue reading up on kstars code to see if it speaks to me ;-)


-- Hans
Last edit: 2 weeks 6 days ago by H__. Reason: spacing
2 weeks 6 days ago #38776

Please Log in or Create an account to join the conversation.

  • Posts: 44
  • Thank you received: 19

Replied by H__ on topic Full Automation and Weather handling in the Scheduler

Hi Ron,

Ron Kramer wrote: I haven't used the scheduler much yet - but this is a necessity for me as well. I THOUGHT the weather watcher was already capable of doing this? IT won't allow me to open if cloudy, but you're saying it won't shut down and park/close?

Exactly. And scary.

I have a script that I run from windows. Is there something similar that I can run from linux?

I see several weather related drivers in INDI, and I added one myself too : Weather Safety Proxy https://indilib.org/devices/weather-stations/weather-safety-proxy.html .

The cloud watcher flags unsafe/clouds or whatever to its data file.
Is there a way to read that file during sub acquisition and abort or wait until (between exposures) to check that file and run a shut down/close up script?

For this you can probably best use the Weather Watcher https://indilib.org/devices/weather-stations/weather-watcher.html as you have several weather measurement values.

I use to leave SGP running and go to bed - no worries. It would always be closed in the morning because even daylight is unsafe. So it would close at dawn.

That's my goal as well. And the weather safety service I wrote (which the weather safety proxy polls) also considers daylight to be unsafe.

So the dat file is written every minute with this format - Really it could look only at SAFE or UNSAFE and close up if unsafe. The rest could be ignored
since unsafe is flagged in the event of any of the other values aren't in the GREEN zone.
"2019-05-03","18:07:21","Cloudy","Dry","Very light","1.2","15.9","4014","1","15.9","10%","21.0","","Closed","5","0","Unsafe","Calm","3.0","Humid","77%"

If you want to only use Safe/Unsafe then using the Weather Safety Proxy does make sense, but you'll have to write a script (or rest API endpoint) that translates the Safe/Unsafe from that dat line to JSON format.

What about watch dog? Can it watch the dat file from the cloud watcher and close up?

It can not afaik. The watch dog only monitors if the indiserver has working clients connected to it (think of a networked INDI setup with indiserver close to the gear and some indi client elsewhere connecting to it over a network).

This was my windows .vbs script file.
dim dome
set dome = CreateObject( "POTH.dome" )
call dome.CloseShutter()
set dome = nothing

The cloudwatcher software would call the script. But I can't run the cloudwatcher software under linux.
I could run the cloudwatcher software on the desktop running Kstars in the house, but it would have to send a command to the nexdome driver to close.

You could use the ekos_sentinel.py for the mean time.

-- Hans
2 weeks 6 days ago #38777

Please Log in or Create an account to join the conversation.

  • Posts: 44
  • Thank you received: 19

Replied by H__ on topic Full Automation and Weather handling in the Scheduler

Hi Ferrante,

fenriques wrote: thanks Hans,
Seems perfect. I will give it a try. Btw I see that we share part of the equipment (10Micron, ASI).

Oh cool !

A few questions:
- the workflow manages the scheduler to start if the weather is ok.

Yes, that's what the scheduler does today (once). And what ekos_sentinel.py does too (continually).

But also resumes a job if after a break the weather returns ok? it seems it doesn't.

When safe weather returns later during the night ekos_sentinel.py will start the scheduler again.

- I guess it should be run with a command like:
<code>python3 ./ekos_sentinel.py 192.168.xxx.xxx --options</code>
Can you provide a simple start command line please?

I use it like this :
./ekos_sentinel.py --indi_host localhost
But there's several settings hardcoded in the script that all need to be parameterized :
INDI_DOME_PARK_PROPERTY = 'Dome Scripting Gateway.DOME_PARK.PARK'
INDI_DOME_PARK_PROPERTY_PARK_SETTING = 'On'
INDI_WEATHER_PROPERTY = 'Weather Meta.WEATHER_STATUS.STATION_STATUS'
INDI_WEATHER_PROPERTY_OK_SETTING = 'Ok'
WEATHER_META_STATION_INDEXES = {1, 2}
INDI_MOUNT_PARK_PROPERTY = '10micron.TELESCOPE_PARK.PARK'
INDI_MOUNT_PARK_PROPERTY_PARK_SETTING = 'On'
INDI_CAMERA_COOLER_PROPERTY = 'ZWO CCD ASI1600MM-Cool.CCD_COOLER_POWER.CCD_COOLER_VALUE'
INDI_CAMERA_COOLER_PROPERTY_OFF_SETTING = '0'
MAIN_LOOP_SLEEP_SECONDS = 60
INDI_COMMAND_SEND_TIMEOUT = 5
MOUNT_PARK_TIMEOUT = 60
ROOF_CLOSE_TIMEOUT = 60

Python's argumentparser provides for some documentation :
./ekos_sentinel.py --help
usage: ekos_sentinel.py [-h] [--debug] [--once] --indi_host INDI_HOST

Ekos Sentinel

optional arguments:
  -h, --help            show this help message and exit
  --debug               enable debug level verbosity
  --once                run only once, useful for debugging
  --indi_host INDI_HOST
                        INDI server address

- For my setup I'll add some time delays (sleep), for example my roof hits the mount if it is not yet parked (I have parking sensors, but better be safe).

My mount can hit the roof too. And I found a bug; the roof starts closing when the mount is still parking :/ I'm glad the 10micron move park fast and my roof is slow :-D There's already a poll loop in there that is supposed to wait for that event. I'll fix that next.

It would be nice to have also a fixed number of retries, e.g. try to warm up the camera for 3 times before issuing a warning.

Makes sense. I'll add that.

-- Hans
2 weeks 6 days ago #38779

Please Log in or Create an account to join the conversation.

  • Posts: 111
  • Thank you received: 23

Replied by fenriques on topic Full Automation and Weather handling in the Scheduler

Hi Ron and Hans,
for sure the Scheduler is not managing shutdown triggered by weather conditions as of today, but the Weather and Dome drivers are managing that.
I tested right now with WeatherWatcher and it works. There's just some configuration to take care of:
- In your Dome driver you should enable the AutoPark switch.
- in your Dome driver enter 'Weather Watcher' (or other driver) in Option->Snoop Devices -> Weather. Think that capital letters and spaces matter.
- Restart Ekos. I think the server doesn't need to be restarted.
- As a test, in Weather Watcher Parameters, lower the temp threshold to a lower value.

Note: this works for the roof driver that I wrote myself (Talon6 RoR) and Weather Watcher. But the AutoPark feature is written in the Dome parent class, so should work for every driver.

Ferrante

EDIT: I know this is a partial solution and not a full shutdown/resume procedure, it just parks the dome. For more advanced solutions there's Hans's python script or the next version of the Scheduler.
Last edit: 2 weeks 6 days ago by fenriques.
2 weeks 6 days ago #38785

Please Log in or Create an account to join the conversation.

  • Posts: 158
  • Karma: 2
  • Thank you received: 40

Replied by sterne-jaeger on topic Full Automation and Weather handling in the Scheduler

H__ wrote: The scheduler class would certainly benefit from refactoring. Is your rework started somewhere yet ? Branch/posts/chat ?

Indeed, but it's in a very early stage. See here: indilib.org/forum/development/5137-parti...-scheduler-code.html

-Wolfgang
2 weeks 6 days ago #38790

Please Log in or Create an account to join the conversation.

Time to create page: 0.713 seconds