I'm checking the indi_maxdomeii driver in order to work ok with Ekos and I have found an issue about security on remote (or automatic) operating of a observatory.

MaxDome II has a nice feature. If it didn't receive any command (like check status) in 120 seconds (or so) it automatically closes the dome shutter. This is very nice in a remote observatory if you loose communication or in case of blackout (Dome shutter is battery powered).

The indi_maxdomeii driver does the job of checking the status once per second to avoid shutter closing. So is driver/computer hangs or can't communicate to the dome, shutter will be closed 2 minutes later.

However, in a remote configuration, if driver is ok but you lose the communication from client to driver, shutter didn't close because driver continues to check status to de dome.

To deal with this situation, indi_maxdomeii driver has a parameter called "Wach Dog Time". If set bigger than zero, it specifies how many seconds will driver wait without any communication with the client before it closes automatically the shutter. This ensures if communication with client is lost, Dome will be closed.

As far as I know, Ekos didn't send any command to the dome regularly, so to work with Ekos "Watch dog time" must be set to zero. However if communication between Ekos and indi_domeii is lost, shutter will remain open.

It would be nice if Ekos could communicate with dome driver every some time (in another client I set "Watch dog time" every second to confirm to the driver the client is up and healthy).

Also I think it will be nice to add "Watch dog time" feature in indidome class, so any dome can have this security feature.

This is a very good point, thank you for bringing it up!

INDI was designed to decouple clients from drivers, so any heart beat signal from the client to the driver would violate this principle. Also, you'd expect ALL clients to follow this behavior (Ekos is not the only client!) and this is not feasible. I think the dome driver should be self-contained. There is "Auto park" feature in INDI::Dome and it can be linked to the weather driver, but in case of total communication loss with the client then it must be tackled differently.

This is not limited to the dome driver, but to all of the observatory equipment. I believe a possible solution is to have a another small client/daemon running on the same PC/Controller that is running the observatory and this Watch dog client/daemon monitors the communication link to the primary remote client. In case of total communication loss, then it should execute a proper shutdown procedure which is probably unique for each observatory.

Actually. putting "Watch dog time" to zero (the default value) disables the feature. So, if a client don't support "Watch dog time" it's ok, it don't need to do.

Yes, it can be interesting to implement this feature as a global one. May be "indiserver" can implement it, and warn drivers whenever communication with client is lost, but i'm not sure if this is the best approach. One observatory can have many indiservers in different computers, and have different clients for the different devices. There is no obligation there exist one integrated client to control all.

On the other hand, every device can have different needs and requiere different reaction times.

In my opinion the most flexible option is to implement this feature for all devices (may be in indidevice.c?), so any device-client pair can have the possibility to control when communication was lost.

The feature will work like this:
- At driver start, "Watch dog time" is zero, and disabled.
- If a client implements it, it can put a value indicating how many seconds of "watch dog time" it needs.
- Client must set this value again before time is over.
- Driver must check if time is over and respond accordingly

You can also have multiple clients connected, which one sends the heart beat? One? All? What if the main client is lost but other clients are still running?

Also, in the case of the dome, it some cases in cannot act alone (for example, park automatically if there is communication loss). Maybe the mount is not yet parked and parking the dome (in case of roll-off) would damage the equipment.

The client who set "Watch dog time" is the responsible to send the heart beat.

Of course, in some cases there is a need for device coordination. In the roll-off case, how is this controlled if user orders to close the roof and the mount is not parked? Or when the whether sensor order to close? I think this case is not different.

If the user directly commands the roll off to close while it poses a danger the mount, it's his/her mistake. But it is automated decisions that should be carefully scrutinized before incorporating them in the driver In case of the weather device, the automatic action is disabled by default as well. However, in this instance, the driver is not coupled to the client, it is snooping from another driver.

You might want to send your proposal to the INDI development mailing list as well to get more opinions on this matter.

One of the nice things about the indi architecture is that we can cascade servers. I've been roughing out how we will manage safety etc for our roll-off, and it will ultimately be based on cascading indi servers like this.

Each telescope will have an indi server running, managing cameras and mounts. These will all be the 'out of the box' servers. Then, for the observatory as a whole, another server will be running, which cascades to each of the physical devices, but, all safety operations will be located there, along this line. There will be a 'mount' driver running, but it will be aware of things like roof status etc. As long as roof status is closed, it will not permit any commands to pass thru to the real mount in the cascade. Once roof status is open, it will become essentially a pass thru device, and allow all commands to pass thru to the real mount. Clients in the house will not connect to the real mount, they connect to the managing server.

Same logic will apply for closing the roof. As long as any of the mounts are not reporting parked position, no commands to close roof will be accepted. Once all of the mounts are reporting parked, then the roof will be allowed to close.

We will take this somewhat further, and there will be means of monitoring the data link between observatory and house. If that link is lost, then the managing server will trigger a sequence of events that ultimately ends up with the roof closed. Weather inputs will be able to trigger the same sequence. Essentially all cameras will have in progress frames aborted, mounts parked, etc. Another input will come from the line power monitors. When line power is not available, then everything will be put into the 'shut down and close up' mode, we will have sufficient battery power available to accomplish a full shut down.

My plan is to have all of the remote safety logic buried in the inteverning server that sits between clients and physical device servers. In the overall design, it's the correct place to put this logic.

All this seems a good design for observatory safety, but one needs to consider what happens if the communication between client, servers or devices fails.
How we detect a communication failure? Clients or devices snooping other device can check if a device is responding, but a device can't do the same with the client.
I think this is a must for a good safety practice. Every device must have the capacity of go to a secure state if it detects a communication issue.

Since shutdown procedure may vary from one observatory setting to another, perhaps the following provides a middle ground:

A simple watchdog INDI driver with one property to set the heartbeat interval, and another for a custom executable to run in case of shutdown?

EDIT: I can also provide a sample Python shutdown script so that users do not need to write one for simple cases. It would have values for:

1. host: default localhost
2. port: default 7624
3. devices list: user provide names for mount, dome...etc
4. shutdown procedure:
4.1 If dome device name is not empty, park dome and wait until park is complete.
4.2 if mount device name is not empty, park mount and wait until park is complete.

The user can then add any custom procedures as desired, but for most cases the script described above would suffice.

That looks to be a good idea. With that way anyone could have a correct shutdown procedure as a script for example, for his/her setup.
Philippe

I put together a small python service that calls a shutdown python script when network is lost. This is just a strawman example i put together to test the concept. I was just starting to look at how to write the shutdown part when i read your post so I'm looking forward to see how that turns out.

I've attached what i worked in case someone else finds it useful. It's been years since I've written any code. This is the first time I've used python as well.

If this worked I was planning to convert to c program. These pythons scripts function on my raspberry pi, but there's not enough error handling at this point. YMMV

(unable to upload the zip file i was trying to attach to this post)

The network itself does not have to be necessarily down, it is the communication link to the client that must be down, or even if the network is still up but the client crashed or was shutdown for whatever reason and the server watchdog is still running.