×

INDI Library v1.9.8 Released (29 Sep 2022)

Bi-monthly INDI Library released with new drivers and bug fixes.

Weather radio don't want connect

  • Posts: 16
  • Thank you received: 3
Adrian, I've been trying to diagnose similar issues as I've been adding in my dew heater code. I haven't solved this but my thoughts are that it could be:

1. Hardware problem - I'm not that neat with my soldering etc and the problems sometimes seem intermittent so I wonder whether it's just something particularly on the i2c bus. I've been gradually redoing my setup, but not sure whether this is the problem.

2. JsonDocument issue - I've been trying to understand JsonDocument to understand whether enough space is being allocated because of string usage and whether we're hitting the memory limit of StaticJsonDocument as more sensors are added. It's a confusing library, but I think the answer here is it's okay.

3. Code problems - I've been through the code in detail, whilst there are some issues in the new update like the missing preprocessor lines you identified, and some missing allocation of space in JsonDocuments, everything looks fine and practically I see similar problems with older versions of the code. So I don't think it's anything that's been recently added.

4. Memory limits of the microcontroller - I notice you're using an Uno same as me. My dew heater code is including math.h as does the Davis library, and I've wondered whether that means there's just not enough left with an Uno when it comes round to allocating memory for the Json strings.

It's been very difficult to narrow it down as it's not particularly repeatable and removing code in various places seems to fix the issue. My guess is it's probably memory, but I'm not an experienced C programmer. It's good to know though that it's not just me :)

-Mark
The following user(s) said Thank You: Adrian
1 year 8 months ago #65674

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
Wolfgang and Mark,

I suspect a memory problem. With three I2C sensors and the Davis sensor
Sketch uses 23780 bytes (73%) of program storage space. Maximum is 32256 bytes.
Global variables use 961 bytes (46%) of dynamic memory, leaving 1087 bytes for local variables. Maximum is 2048 byte

This is why it works when I use the Davis on its own, and not when I add more sensors.

I don't understand the Static Json docsize declaration size based upon JSON_OBJECT_SIZE (will have to read up on it), it
can't know what the lenght of the variables are and what strings they will hold, right!
The script creates this then serializes this into a string which is returned.

My w json serialized string is 313 bytes long!
{"Davis Anemometer":{"init":true,"direction":96,"avg speed":0,"min speed":0,"max speed":0,"rotations":0},"BME280":{"init":true,"Temp":12.25,"Pres":1024.765,"Hum":52.88281},"MLX90614":{"init":true,"T amb":12.45001,"T obj":12.91},"TSL2591":{"init":true,"Lux":2.062501,"Visible":799,"IR":211,"Gain":32,"Timing":1}

If I type p I get strange results, but with w it can look right if I fiddle with the docSize.
Weather Radio V 1.10
{"Davis Anemometer":{"init":true,"direction":96,"avg speed":0,"min speed":0,"max speed":0,"rotations":0},"BME280":{"init":true,"Temp":12.25,"Pres":1024.765,"Hum":52.88281},"MLX90614":{"init":true,"T amb":12.45001,"T obj":12.91},"TSL2591":{"init":true,"Lux":2.062501,"Visible":799,"IR":211,"Gain":32,"Timing":1}}
{
  "Davis Anemometer": {
    "init": true,
    "direction": 96,
    "avg speed": 0,
    "min speed": 0,
    "max speed": 0,
    "rotations": 0
  },
  "BME280": {
    "init": true,
    "Temp": 12.25,
    "Pres": 1024.765,
    "Hum": 52.88281
  },
  "MLX90614": {
    "init": true,
    "T amb":
    "Visible": 799,
    "IR"iming": 1
  }
}
Again note that the last line in pretty is taking the start of the IR for the MLX sensor and ending it with the last field of the TSL sensor.

My impression is lack of memory on the microcontroller.

Adrian
1 year 8 months ago #65700

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
Seems docSize needs to include the json part and the string storage.

github.com/bblanchon/ArduinoJson/issues/151

For my 313 byte serial output the doc size is calculated as 152 so it is unlikely to be able to hold the information.
Weather Radio V 1.10
152 <=== Serial.println(docSize)
{
  "Davis Anemometer": {
    "init": true,
    "direction": 87,
    "avg speed": 0,
    "min speed": 0,
    "max speed": 0,
    "rotations": 0
  },
  "BME280": {
    "init": true,
    "Temp": 14.42,
    "Pres": 1024.433,
    "Hum": 50.88184
  },
  "MLX90614": {
    "init": true,
    "T amb": 14.82999,
    "T obj": 14.19
  },
  "TSL2591":<Rni
  }
}
1 year 8 months ago #65704

Please Log in or Create an account to join the conversation.

  • Posts: 985
  • Thank you received: 297
Ah, interesting, that issue may explain it. I had the same problem with getCurrentConfig()
String getCurrentConfig() {
  const int docSize = JSON_OBJECT_SIZE(7) + // max 7 configurations
                      JSON_OBJECT_SIZE(2) + // DHT sensors
                      JSON_OBJECT_SIZE(3) + // Davis Anemometer
                      JSON_OBJECT_SIZE(1) + // Water sensor
                      JSON_OBJECT_SIZE(2) + // Rain Sensor
                      JSON_OBJECT_SIZE(3) + // WiFi parameters
                      JSON_OBJECT_SIZE(1) + // Arduino
                      JSON_OBJECT_SIZE(3) + // OTA
                      JSON_OBJECT_SIZE(2);  // buffer
  StaticJsonDocument <docSize> doc;
...
As you can see, I simply added an additional buffer of JSON_OBJECT_SIZE(2) - I found it out by trial and error...

Wolfgang
TSA-120 + FSQ-85 + ONTC 10"F4 Newton (+ epsilon-160 on Japan trip) | Avalon Linear + M-zero | ASI 1600mm pro + 6200mm pro | KStars/INDI on Raspberry Pi 4/Intel NUC
1 year 8 months ago #65710

Please Log in or Create an account to join the conversation.

  • Posts: 16
  • Thank you received: 3
This is the best description I found of how to use the library: arduinojson.org/v6/how-to/determine-the-...of-the-jsondocument/ My reading of this was that if all strings used for keys are const char* then we won't need extra space, otherwise serialisation needs space to copy all the strings.

-Mark
1 year 8 months ago #65712

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
Slight progress

There is a clue to setting the Static JsonDocument in the webpages

arduinojson.org/v6/assistant/

I set mine to 194 to allow for the JSON data and the strings
How to work this out on the fly for different sensor and outputs?

The main problem with the outputting is with serializeJsonPretty, this puts out a very long output with all those nice spaces and ^M at the end.
This seems to overrun the memory which is put into String result="";
I am not sure how the String object looks after its memory, or if you have to reserve memory for it yourself, and free it up afterwards etc.

You can specify how much to output in the serialze functions

I tried to output to a buffer and set a size limit on this
#include <ArduinoJson.h>
#define DESTBUFSIZE 320
char destBuf[DESTBUFSIZE];
 
#ifdef __arm__
// should use uinstd.h to define sbrk but Due causes a conflict
extern "C" char* sbrk(int incr);
#else  // __ARM__
extern char *__brkval;
#endif  // __arm__
 
int freeMemory() {
  char top;
#ifdef __arm__
  return &top - reinterpret_cast<char*>(sbrk(0));
#elif defined(CORE_TEENSY) || (ARDUINO > 103 && ARDUINO != 151)
  return &top - __brkval;
#else  // __arm__
  return __brkval ? &top - __brkval : &top - __malloc_heap_start;
#endif  // __arm__
}
 

In the getSensorData I changed it to use the buffer and permitted length.
void getSensorData(bool pretty) {
 
and
 
  int ret;
 
  destBuf[0] = 0;// Null terminator
 
  if (pretty)
    ret = serializeJsonPretty(weatherDoc, destBuf, DESTBUFSIZE);
  else
    ret = serializeJson(weatherDoc, destBuf, DESTBUFSIZE);
 
  Serial.print("ret :"); Serial.println(ret);
  Serial.print("Free Memory: "); Serial.println(freeMemory());
  Serial.print("DestBuf :"); Serial.println(destBuf);

Then in the w and p commands
    case 'w':
      //Serial.println(getSensorData(false));
      getSensorData(false);
      Serial.println(destBuf);
      break;
    case 'c':
      Serial.println(getCurrentConfig());
      break;
    case 'p':
      //Serial.println(getSensorData(true));
      getSensorData(true);
      Serial.println(destBuf);
      break;

Not sure if this is the best approach. I do remember trying to understand the firmata stuff on the old meteostation and wringing my hands.
1 year 8 months ago #65715

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
Of course the Wemos D1 mini (4Mb) pro (16Mb) is flush with memory compared to the Uno (32K flash and 2K SRAM)

So if a Wemos D1 mini pro is used there will be no shortage of ram compared to the UNO.
1 year 8 months ago #65753

Please Log in or Create an account to join the conversation.

  • Posts: 985
  • Thank you received: 297
Adrian,
I filled a breadboard with a BME280, a DHT11, a MLX90614, a TSL2591, a RG-11, a Davis Wind sensor and a OLE display. At least on a Wemos D1 mini, serializing works fine:
{
 
  "Davis Anemometer": {
    "init": true,
    "direction": 3,
    "avg speed": 0,
    "min speed": 0,
    "max speed": 0,
    "rotations": 0
  },
 
  "Rain Sensor": {
    "init": true,
    "rainfall": 0,
    "count": 0
  },
 
  "BME280": {
    "init": true,
    "Temp": 21.08,
    "Pres": 974.0237,
    "Hum": 26.83203
  },
 
  "DHT": {
    "init": true,
    "Temp": 19.9,
    "Hum": 36
  },
 
  "MLX90614": {
    "init": true,
    "T amb": 21.01,
    "T obj": 20.55001
  },
 
  "TSL2591": {
    "init": true,
    "Lux": 1.487565,
    "Visible": 505,
    "IR": 108,
    "Gain": 32,
    "Timing": 1
  }
}

Therefore I suspect it is a memory issue or a cabling problem.

Wolfgang
TSA-120 + FSQ-85 + ONTC 10"F4 Newton (+ epsilon-160 on Japan trip) | Avalon Linear + M-zero | ASI 1600mm pro + 6200mm pro | KStars/INDI on Raspberry Pi 4/Intel NUC
1 year 8 months ago #65806

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
Thanks for looking at that Wolfgang, you are very kind to do all that work.

Do you have an Uno to try with?

Like Mark I doubted my wiring so I had redone it several times before I looked at the code.

Yes, it definitely seems to be due to lack of memory on the UNO.

Options:
- Use bigger memory device, like your Wemos D1
- Limit sensors used
- Spread sensors over several microcontroller boards
- It may be possible to save some memory, so that lower powered devices can be used

The main memory hogs are the JSON calls that are copied back to a large long string. The JSON object is big and the string output to the serial port is larger.
In addition, reading about the String object suggests that it is somewhat notorious for giving memory problems.

I have been looking at the code and I am trying to get rid of the String objects and the ArduinoJSON library.

I see the same information is in three places - the basic structure storing the results for each sensor, the JSON object
and the large character buffer that is built up to send to the serial port.

I have written a shortJSON library to generate the JSON strings. I store these in a character buffer for each sensor (the memory is assigned at the start and continuously reused) I just allocated a little more characters than are needed to accomodate the sensor readings.
At the moment I call the function to update this character buffer every time
when the sensor is updated (I call it after the structure is updated eg mlxData.variable.= mlxreadCAll(object).
This is a little wasteful in processing time but the strings are just there and ready to go when they are requested.

These are just Serial.print(MLXstring); on the calls to w, p etc. from the serial console.

I have these working for davis, mlx, tsl and bme, no problem with funny output or memory shortage so far, so seems stable with less memory usage.

Before I move on to the other sensors I will see if I can get indi to talk to the new code.

I will upload the code to github so you can have a look at it.

Kind regards,

Adrian
1 year 8 months ago #65811

Please Log in or Create an account to join the conversation.

  • Posts: 985
  • Thank you received: 297
Is there a specific reason why you are using an Uno instead of a ESP8266 type Arduino? A Wemos D1 mini looks like significantly more powerful, has WiFi inside, is quite small and costs only a few Euros...
TSA-120 + FSQ-85 + ONTC 10"F4 Newton (+ epsilon-160 on Japan trip) | Avalon Linear + M-zero | ASI 1600mm pro + 6200mm pro | KStars/INDI on Raspberry Pi 4/Intel NUC
1 year 8 months ago #65859

Please Log in or Create an account to join the conversation.

  • Posts: 16
  • Thank you received: 3
Personally, I have lots of unos and xbees lying around so I'm just trying to use them rather than buying new kit. There's something nice too about using minimal kit - the easy option for these kind of projects is always to use another raspberry pi and wireless - the cost isn't too much, but where's the fun in that? :)
1 year 8 months ago #65860

Please Log in or Create an account to join the conversation.

  • Posts: 89
  • Thank you received: 16
I have unos and nanos and no Wemos D1, probably should invest I suppose.

I have been using the usb cable to power the microcontroller and communicate with it (two objectives with one cable).

I bought a longer USB cable which goes to the weather station's microcontroller.
This is commected back to a raspberry pi in a nicer weather proof environment which has AC power and an ethernet connection.

The wifi from the house does not reach the garden well where the pi is and is much less reliable than the cable usually.

I did set up the pi as a wifi hotspot so that the pi zero on the skycam can communicate back, so it would be easy to use this trick with a Wemos.

Still have to get power to the pizero and have used a portable power bank 5V (lasts the whole night) and 12V DC via buck converter to 5 V,
which I can power via a 12V battery or a 12V power supply running off the mains. Suppose I could power the Wemos in a similar fashion.

Bit old fashioned with the uno, I know. Suppose I should move with the times!
1 year 8 months ago #65862

Please Log in or Create an account to join the conversation.

Time to create page: 1.061 seconds