Exception 0: Illegal instruction
PC: 0x4020207c
EXCVADDR: 0x00000000

Decoding stack results
0x40100776: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring_digital.cpp line 142
0x40100808: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/interrupts.h line 29
0x40100d90: check_poison_block(umm_block*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 82
0x40100e25: check_poison_neighbors(umm_heap_context_t*, uint16_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 69
0x4020208c: displayTSL2591Parameters() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x401001e2: isr_rg11_rain_event() at C:\Users\minhlead\Downloads\weatherradio/rg11.h line 26
0x4010081c: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring_digital.cpp line 175
0x40100808: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/interrupts.h line 29
0x40100d90: check_poison_block(umm_block*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 82
0x40100288: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 181
0x4023fd29: umm_info_safe_printf_P(char const*, ...) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 207
0x4020df8a: Print::printf(char const*, ...) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Print.cpp line 62
0x40100288: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 181
0x40100d90: check_poison_block(umm_block*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 82
0x40100de0: check_poison_neighbors(umm_heap_context_t*, uint16_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 52
0x40101078: umm_malloc_core(umm_heap_context_t*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 655
0x40100f31: umm_free_core(umm_heap_context_t*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 541
0x40101534: umm_realloc(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 1160
0x40100c78: get_poisoned(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 113
0x40101575: umm_poison_realloc_fl(void*, size_t, char const*, int) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 16
0x40100858: calloc(size_t, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\heap.cpp line 252
0x40100f31: umm_free_core(umm_heap_context_t*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 541
0x40101534: umm_realloc(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 1160
0x40100c78: get_poisoned(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 113
0x40100f31: umm_free_core(umm_heap_context_t*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 541
0x40101534: umm_realloc(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 1160
0x40100c78: get_poisoned(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 113
0x40101575: umm_poison_realloc_fl(void*, size_t, char const*, int) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 16
0x40100858: calloc(size_t, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\heap.cpp line 252
0x4020cdc0: ArduinoOTAClass::onStart(std::function) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\libraries\ArduinoOTA\ArduinoOTA.cpp line 45
0x4020d300: EspClass::getFlashChipId() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Esp.cpp line 278
0x4020ce57: ArduinoOTAClass::onProgress(std::function) at c:\users\minhlead\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.0.0-newlib4.0.0-gnu23-48f7b08\xtensa-lx106-elf\include\c++\10.2.0\bits/std_function.h line 587
0x4020d215: ArduinoOTAClass::_runUpdate() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/IPAddress.h line 166
0x40204996: esp8266webserver::ESP8266WebServerTemplate::_parseRequest(WiFiClient&) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x4020d300: EspClass::getFlashChipId() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Esp.cpp line 278
0x40204ecd: esp8266webserver::ESP8266WebServerTemplate::_prepareHeader(String&, int, char const*, unsigned int) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x402017a4: S2Stream::read() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 181
0x40205f3c: ArduinoJson6180_91::TextFormatter >::writeFloat(float) at C:\Users\minhlead\Documents\Arduino\libraries\ArduinoJson\src/ArduinoJson/Json/TextFormatter.hpp line 143
0x40100288: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 181
0x4022273b: nan at /workdir/repo/newlib/newlib/libm/common/s_nan.c line 44
0x40223813: __sfvwrite_r at /workdir/repo/newlib/newlib/libc/stdio/fvwrite.c line 124
0x40100d90: check_poison_block(umm_block*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 82
0x40100e25: check_poison_neighbors(umm_heap_context_t*, uint16_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 69
0x40101078: umm_malloc_core(umm_heap_context_t*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 655
0x40100de0: check_poison_neighbors(umm_heap_context_t*, uint16_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 52
0x40100c78: get_poisoned(void*, size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_poison.c line 113
0x40101298: umm_malloc(size_t) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc\umm_malloc.cpp line 794
0x401005fd: millis() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring.cpp line 193
0x4020208c: displayTSL2591Parameters() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x401001e2: isr_rg11_rain_event() at C:\Users\minhlead\Downloads\weatherradio/rg11.h line 26
0x4010081c: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring_digital.cpp line 175
0x40100808: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/interrupts.h line 29
0x40100288: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 181
0x4020ebbe: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 473
0x40107135: cont_can_yield(cont_t*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0/tools/sdk/include/ets_sys.h line 113
0x4020eba8: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 473
0x4020208c: displayTSL2591Parameters() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x4010081c: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring_digital.cpp line 175
0x401001e2: isr_rg11_rain_event() at C:\Users\minhlead\Downloads\weatherradio/rg11.h line 26
0x4010081c: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring_digital.cpp line 175
0x40100808: interrupt_handler(void*, void*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/interrupts.h line 29
0x40100288: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 181
0x40107132: cont_can_yield(cont_t*) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0/tools/sdk/include/ets_sys.h line 113
0x4020eba8: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 473
0x4020eb2d: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 442
0x4020ee59: UpdaterClass::printError(Print&) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 555
0x4020eb2d: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 442
0x4020ee59: UpdaterClass::printError(Print&) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 555
0x4020eb2d: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 442
0x4020eb2d: UpdaterClass::_verifyEnd() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 442
0x4020ee59: UpdaterClass::printError(Print&) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 555
0x4020ee59: UpdaterClass::printError(Print&) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Updater.cpp line 555
0x401005fd: millis() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_wiring.cpp line 193
0x40206d54: updateDisplayText() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/WString.h line 79
0x4020e010: Print::printf_P(char const*, ...) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Print.cpp line 75
0x4020f092: String::changeBuffer(unsigned int) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\WString.cpp line 207
0x40206a96: displayWiFiParameters() at C:\Users\minhlead\Downloads\weatherradio/esp8266.h line 310
0x401002a9: esp_schedule() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 128
0x4020e13c: Print::print(char) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\Print.cpp line 125
0x4010158d: umm_poison_realloc_fl(void*, size_t, char const*, int) at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_local.c line 143
0x40100090: app_entry() at C:\Users\minhlead\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\core_esp8266_main.cpp line 328

Okay so above is what the serial monitor spewing after exception decode. The first line refer to the below line of code from core_esp8266_wiring_digital.cpp line 142 (I colored the line in red)

void IRAM_ATTR interrupt_handler(void *arg, void *frame)
{
  (void) arg;
  (void) frame;
  uint32_t status = GPIE;
  GPIEC = status;//clear them interrupts
  uint32_t levels = GPI;
  if(status == 0 || interrupt_reg == 0) return;
  ETS_GPIO_INTR_DISABLE();
  int i = 0;
  uint32_t changedbits = status & interrupt_reg;
  while(changedbits){
    while(!(changedbits & (1 << i))) i++;
    changedbits &= ~(1 << i);
    interrupt_handler_t *handler = &interrupt_handlers;
    if (handler->fn && 
        (handler->mode == CHANGE || 
         (handler->mode & 1) == !!(levels & (1 << i)))) {
          // to make ISR compatible to Arduino AVR model where interrupts are disabled
          // we disable them before we call the client ISR
          esp8266::InterruptLock irqLock; // stop other interrupts
          if (handler->functional)
          {
              ArgStructure* localArg = (ArgStructure*)handler->arg;
              if (localArg && localArg->interruptInfo)
              {
                  localArg->interruptInfo->pin = i;
                  localArg->interruptInfo->value = __digitalRead(i);
                  localArg->interruptInfo->micro = micros();
              }
          }
          if (handler->arg)
          {
              ((voidFuncPtrArg)handler->fn)(handler->arg);
          }
          else
          {
              handler->fn();
          }
      }
  }
  ETS_GPIO_INTR_ENABLE();
}

Hope someone with better understanding of code can shed some light here?

Okay so a quick google search it seems this article is describing what I am experiencing (except for the board crash while connecting to a Wifi hotspot, not when trying to establish a connection)
www.mikrocontroller.net/attachment/37839...ernal_Interrupts.pdf
How do I put this into the code and try again?

Are you familiar with GIT? I could create a new branch on my indi-3rdparty fork so that you could download the latest firmware version and install it on your ESP8266.

Wow that was fast. I am familiar with how github work. Please fork it and send me the link, I know how to proceed. Thanks!

I found a place in the rain sensor code that wasn't protected properly as interrupt. It would be great if you give it a try. It's branch weatherradio-v_1.16 at my indi-3rdparty clone github.com/sterne-jaeger/indi-3rdparty

Alright, it seems the code WORKED! I let it ran for straight 10 minutes with the fast switching relay (5 click per secs) and it does not crash like it used too (it took 10-20secs before your fix for the board to gave up). Let me try letting it run for a few hours and see what happens. Thank you so much!
Also, I can see that the "count" var in the rainsensor.h is an unsigned Interger (which will display the max value to 65k ish). And the count does not reset, it just keep adding up since the board boots up. What would happens if the counts get to 65k (which is quite frankly, not that much since a heavy rain here in Vietnam can have somewhere between 10-20k clicks, so 65k is like 4-5 big rain which is a week worth of rain on raining season here).
 

Okay I think you fixed it. Thanks a heap! This got me scratched my head for weeks now.

OK, the new version 1.16 has found its way into the master and is available now in the latest sources at github.com/indilib/indi-3rdparty

Btw: the count variable is only used for bucket mode. For drop detection the frequency is calculated by intervalCount, which is typically reset to 0 once a minute. And for the bucket mode, I would question that anyone sees the overflow.

Umh. I am using drop detection mode and the count does not resets. It just keep adds up.the drop rate is reset every cycle though

For simplicity reasons the count variable is delivered for both types of rain sensors, bucket count and drop count. For the latter, only the eventFrequency is of relevance, count can be ignored.

Thanks. That means even if the counter overflows the board wont crash right?
about the false positive that you said before, check to see if you are having something that cast rotating shadows over your rg11? mine have some false clicks when I put it under the annonemeter so the blades cast rotating shadows over the sensor. When I move it to clearer sky, the false alarm stops.
but mine was just some random click, not a series of click like yours. Is your cable from your rg11 to the esp long? If the cable is long, it should be in twisted pair otherwise it could be interference.