1. 14
  1. 1

    The really resource-constrained way to do this would be to remember “SAX vs. DOM” from the 90s and take the SAX route. Never accumulate the API response body, instead stream it into an evented JSON parser that runs callbacks when it completes parsing a node. The callback can look at the path, ignore data it doesn’t care about, and accumulate the values of the two time series it does care about into arrays. Those plus a stack for the parser to remember where it is are all that ever needs to live long.

    A really nice parser of this type would even have a way to tell it that you’re only interested in events underneath a certain path – not only can it avoid making callbacks for irrelevant data, it can put the parser into a “skipping” mode where it runs faster and doesn’t even bother allocating space for data that’s only going to get thrown away. It just has to pay enough attention to ensure that it goes back into regular mode at the right moment.

    I don’t know if such a parser exists for JSON, in Python… it could, but operating this way is generally seen as more trouble than it’s worth when 4GB of RAM is in “here’s a nickel, buy yourself a real computer” territory :)

    1. 1

      Yeah, absolutely. A streaming parser for JSON doesn’t exist for MicroPython, it might for regular Python.

      I did actually start writing a character oriented parser for the constrained format of JSON that InfluxDB returns (much simpler than a SAX parser for all of JSON). However, then I realised that json.load() was already working fine, and the query would never ask for more than 1 hour of samples so there was an upper limit on the size of the JSON document it returned.

      1. 1

        Yeah. For what it’s worth, I’m not criticizing at all, this is pretty cool :)

        I recently took my first venture into solar myself, but it’s not running my house — just a couple little panels sitting in my backyard feeding an APRS repeater with an antenna up in a tree. The radio and raspberry pi draw about 7W combined on average, and the battery is good for 480Wh so it will stay up for about 2-3 days of bad weather (it’s not hooked up to utility power at all unless I bring the battery inside and charge it). And I can send telemetry from the battery charger using… you got it, APRS :)