utilities/telegraf-plugins#9: Soliscloud Plugin



Issue Information

Issue Type: issue
Status: closed
Reported By: btasker

Created: 13-May-23 09:03



Description

I've got solar panels being installed soon, along with a Ginlong Solis inverter.

It'll report generation stats into SolisCloud.

The platform has an API (although you need to request access), so I want to put together an exec plugin to periodically pull generation stats from the API.

Obviously, that'll be easier to do once the inverter's installed, but there's an API Doc so I'd like to see if we can hit the ground running.

The API docs seems to note that rate limits are quite strict

Note: The calling frequency of all interfaces is limited to three times every five seconds for the same IP

So the plugin will need to account for that



Toggle State Changes

Activity


Their API uses signature based authentication - using a shared secret to generate a HMAC of the request specifics. It's a little unusual in that they're using base64'd hashes rather than hexdumps, but it's otherwise relatively straightforward.

The API doc defines the auth header structure as

Authorization = "API " + KeyId + ":" + Sign
Sign = base64(HmacSHA1(KeySecret,
VERB + "\n"
+ Content-MD5 + "\n"
+ Content-Type + "\n"
+ Date + "\n"
+ CanonicalizedResource)

It doesn't give a clear specification of what the expected behaviour is for Content-MD5 if the request body is empty:

When the message content is empty, the string is empty.

I've taken that to mean that the MD5 should be omitted (i.e. that part of the signstring should just be `\n") rather than that the input to the digest should be an empty string.

So, once I've got access, we may need to come back and change that (assuming we generate any requests with empty request bodies in the first place).

Commit dc32216 implements support for the auth mechanism

changed the description

v1/api/userStationList

The API doc and the cloud interface use slightly different terminology, but it looks like v1/api/userStationList will give a list of what the Cloud UI call "Plants" (i.e. locations where inverters are installed) - in my case, there'll be just one listed (though a business might have more).

It gives an overview of total solar yield, battery charge rates and whether there are any faults detected - obviously all useful things to collect stats from

For this endpoint (and I'm guessing all others) there's a compulsory request body

{"pageNo":1,"pageSize":100}

/v1/api/stationDetail

This endpoint provides more detailed information about the location, including cumulative counters (such as money saved etc).

Depending on what actually gets populated once the device is installed, this will totally be worth calling.

It has one required request body property

{"id":1234}

(where that ID is taken from the station list)

But, it'll be worth being conscious of rate limits - I won't hit them (because only one location), but if someone else with multiple locations were to use this plugin they might - when iterating over stations listed in /v1/api/userStationList will need to be conscious of that

/v1/api/inverterList

Lists installed inverters, like the station listing, it requires

{"pageNo":1,"pageSize":100}

/v1/api/inverterDetail

Gives extended detail on a specific inverter, including batter details.

Requires either the ID, or the serial number to be provided

{"id":1234, "sn": "abcdef"}
verified

mentioned in commit github-mirror/telegraf-plugins@dc322167dc07caa75d4b4e240b5eb7add3c148c2

Commit: github-mirror/telegraf-plugins@dc322167dc07caa75d4b4e240b5eb7add3c148c2 
Author: B Tasker                            
                            
Date: 2023-05-13T10:03:35.000+01:00 

Message

Implement support for Solis's authorization mechanism (see utilities/telegraf-plugins#9)

It returns the correct values for the examples given in their docs, but I'm not currently able to test this against the API: I've not been sent creds yet (in fact, my inverter's not even been installed yet)

It's a fairly simple signature mechanism, so there shouldn't be much to go wrong with it - the most likely cause of issues will be around the request body (the doc says that the hash should be empty if the body is empty, but doesn't clarify whether they mean it should be omitted, or a hash of a null string. The example in the doc has a request body).

+130 -0 (130 lines changed)

/v1/api/alarmList

Lists details of alarms/faults.

Returns the station ID, the serial number of the alarming device along with alarm codes and levels.

There's a list of alarm codes [here](https://oss.ginlong.com/templet/Alarm%20information%28%E6%8A%A5%E8%AD%A6%E4%BF%A1 %E6%81%AF%29.xls)

Input is

{"pageNo":1,"pageSize":100, "stationId": 1234}

Station ID is optional

The API has a number of other endpoints - some are for collecting historic data, others relate to grabbing data from data-loggers and Electronic Power Monitoring (EPM) boxes: I'm not having those installed, so won't add support for those.

Need to think about the eventual schema then.

My existing power consumption data goes into a measurement called power_watts. That could be reused as long as we avoid existing field names (to ensure that the new data doesn't lead to existing graphs misreading) - I'll likely make the measurement name configurable anyway.

Essentially though, we've got one set of information:

  • Generated
  • Money saved
  • Battery charge %age
  • Battery discharge rate
  • Battery discharge today
  • Battery charge added today
  • Alarm state/code/level

But the data is available at different scopes - per device/inverter and per-site.

So, we probably want to convert to a single schema, with tags differentiating whether this is scoped to a site or a device.

power_watts,type=device,device_type=battery,inverter_id=1234 charge=99.9,output=300,today_charge=500,today_consumed=100
power_watts,type=device,device_type=inverter,inverter_id=1234,alarm_state=0 today_generated=300,today_money=3.50,alarm_code=3

That's obviously going to change a little depending on what results the API actually gives (the API doc doesn't give much detail/example).

verified

mentioned in commit github-mirror/telegraf-plugins@b1d1b5f587aed2a152d922a9fd40dcda6a915b32

Commit: github-mirror/telegraf-plugins@b1d1b5f587aed2a152d922a9fd40dcda6a915b32 
Author: B Tasker                            
                            
Date: 2023-05-14T12:25:09.000+01:00 

Message

Implement support for rate-limit observance (utilities/telegraf-plugins#9)

This adds a ratelimit property to the object which is used to keep track of the number of requests placed and the time that we started counting.

A rough overview is

  • Every quota period (5s by default) the timer is reset.
  • If there are more than n (3 by default) requests in the quota period, the request must wait

If a request is waiting, the quota will be rechecked every 1 second up to a maximum of max_ratelimit_wait seconds (default 8). If the limit is hit, the script will exit with exit code 1.

With the default config max_ratelimit_wait should never actually be hit and exists only as a safety net against me writing in a bug which causes an infinite loop

It might be mildly interesting to have the script's LP output also provide details on its behaviour around quotes (how often and how long it has waited etc)

+89 -5 (94 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@374ffd53b510b89d1619df05259a76c66e2523d8

Commit: github-mirror/telegraf-plugins@374ffd53b510b89d1619df05259a76c66e2523d8 
Author: B Tasker                            
                            
Date: 2023-05-14T11:51:21.000+01:00 

Message

Add support for stationList (utilities/telegraf-plugins#9)

This implements support for the first API call, as well as loading config from some environment variables:

  • API_ID
  • API_SECRET
  • API_URL
+57 -0 (57 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@8bc3e47fc0804b28939805a8e6abbbb69a84faeb

Commit: github-mirror/telegraf-plugins@8bc3e47fc0804b28939805a8e6abbbb69a84faeb 
Author: B Tasker                            
                            
Date: 2023-05-14T13:42:09.000+01:00 

Message

Add a mocked response to fetchStationList() (utilities/telegraf-plugins#9)

I realised that the Cloud UI makes calls that look very similar to the API doc's description, so to help proceed with implementing the next sets of api calls, have added a mocked response - these will need to be removed before release

+43 -2 (45 lines changed)

To help progress development, I've added mocked responses to the script - they'll need removing once devices are reporting into the API.

Currently the script:

  • Calls stationList to get a list of locations
  • Calls inverterList to get a list of inverters in each location
  • Calls inverterDetail to get battery information from each inverter

None of this, yet, gets turned into Line Protocol, so that's probably the next thing to do.

verified

mentioned in commit github-mirror/telegraf-plugins@5c032a3308c3d79740fe88ddd24699e29b13c9ed

Commit: github-mirror/telegraf-plugins@5c032a3308c3d79740fe88ddd24699e29b13c9ed 
Author: B Tasker                            
                            
Date: 2023-05-14T14:02:58.000+01:00 

Message

Collect details of deployed inverters (utilities/telegraf-plugins#9)

This calls the inverterList endpoint to get details of inverters at a given site.

+114 -21 (135 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@eb21e04a5d85ff72d3ad980e9728f03969ef5768

Commit: github-mirror/telegraf-plugins@eb21e04a5d85ff72d3ad980e9728f03969ef5768 
Author: B Tasker                            
                            
Date: 2023-05-14T14:42:12.000+01:00 

Message

Extract battery usage statistics and generate line protocol to expose the stats (utilities/telegraf-plugins#9)

This primarily uses the names exposed by the upstream API - once we have a more definitive idea of what each column exposes it may be worth giving more descriptive names.

This commit also adds a calculated field: readingAge

This field is an integer describing the time (in seconds) between the data being reported to the API (i.e. dataTimestamp) and the reading being collected by the script.

It should allow us to build an alert to fire if the inverter stops reporting in

+61 -4 (65 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@6a46ce0852d6763c489a7f4e4615ef248b0cd0d7

Commit: github-mirror/telegraf-plugins@6a46ce0852d6763c489a7f4e4615ef248b0cd0d7 
Author: B Tasker                            
                            
Date: 2023-05-14T14:13:14.000+01:00 

Message

Add call to inverterDetail (utilities/telegraf-plugins#9)

The API doc is particularly vague for this path, and the UI isn't currently returning any data from it's equivalent call. This will almost certainly need revisiting once my inverter's in.

Note: The API doc actually says bstteryCurrent not batteryCurrent, I've assumed this is a typo in the doc (although I have seen some typos in the Cloud UI's response bodies)

+64 -2 (66 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@cbea4b4d14506fef59df8bb9bc5beb307edea060

Commit: github-mirror/telegraf-plugins@cbea4b4d14506fef59df8bb9bc5beb307edea060 
Author: B Tasker                            
                            
Date: 2023-05-14T14:55:23.000+01:00 

Message

Translate inverter details into LP (utilities/telegraf-plugins#9)

+52 -2 (54 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@c55191e629c1ed232ffc9333882e4ab87c168bf1

Commit: github-mirror/telegraf-plugins@c55191e629c1ed232ffc9333882e4ab87c168bf1 
Author: B Tasker                            
                            
Date: 2023-05-15T09:01:59.000+01:00 

Message

Bugfix: Switch signstr to using literal newlines

They did want full newlines after all.

First successful call to the API has been made (utilities/telegraf-plugins#9)

+2 -3 (5 lines changed)

changed the description

verified

mentioned in commit github-mirror/telegraf-plugins@f6642d24077f6ad6f762421c04f960c54a40d7fd

Commit: github-mirror/telegraf-plugins@f6642d24077f6ad6f762421c04f960c54a40d7fd 
Author: B Tasker                            
                            
Date: 2023-05-18T08:43:41.000+01:00 

Message

On a re-read of the API docs (rather than a skim), all responses are going to have the nested structure (utilities/telegraf-plugins#9)

{ "success": true, "code": "0", "msg": "Successful", "data": { }, }

TODO:

  • Should probably simplify the conditionals by checking that structure for success.
  • I'd also like to move the API response specific checks into the methods themselves rather than the caller needing to do it
+36 -28 (64 lines changed)

Now that the install's complete, I've made some tweaks based on observation of the API output.

There are some headaches with the API's output though

  • The value of gridPurchasedTodayEnergy is (according to gridPurchasedTodayEnergyStr) in kWh. Yet it's claiming ~650, suggesting it's actually in Wh
  • That 650 value, though, doesn't match anything in their Cloud UI/app

If it's simply mislabelled and is consistently in Wh, that's fine. My concern is whether it switches to being in kWh once it reaches over 1000 (note: it does - 24h later I'm getting a reading of 5.04kWh - we might need to guess unit based on some sort of threshold check).

Extra Notes:

  • Each of the panel array's current power can be found in pow1, pow2 etc. These are in Wh.
  • Despite what the API docs say, BatteryPowerPec does not seem to give a percentage for the battery. batteryCapacitySoc however, does.
  • The station object also contains some interesting information: capacityPercent details what percentage of the installed solar capacity is currently being realised. condTxtD gives a cloudy/sunny etc

The API docs say that field state has the following values/meanings

Screenshot_20230601_140722

However, my inverter is currently reporting state as 3 (so, according to the above, Alarm).

But, the Cloud UI doesn't show anything in an alarm state Screenshot_20230601_140828

I'm going to leave collection of that value in the plugin, but if alerting is built around it it's probably better to look for changes rather than relying on the values in the docs.

verified

mentioned in commit github-mirror/telegraf-plugins@10166fc200913156801463aacc986d2d08409dcd

Commit: github-mirror/telegraf-plugins@10166fc200913156801463aacc986d2d08409dcd 
Author: B Tasker                            
                            
Date: 2023-06-01T13:50:01.000+01:00 

Message

Add stats on installed capacity and percentage generated (utilities/telegraf-plugins#9)

This adds two new fields to the inverter device type:

  • stationCapacity: the installed capacity (in kWp)
  • stationCapacityUsedPerc: the percentage of that capacity currently being delivered by the panels
+6 -4 (10 lines changed)
verified

mentioned in commit github-mirror/telegraf-plugins@7975d879010545a1feaea020b0b6f69c165019b1

Commit: github-mirror/telegraf-plugins@7975d879010545a1feaea020b0b6f69c165019b1 
Author: B Tasker                            
                            
Date: 2023-06-01T14:02:57.000+01:00 

Message

Add total consumption (utilities/telegraf-plugins#9)

This adds field consumptionToday to the inverter stats.

The figure represents total consumed, from all sources (i.e. solar, battery, grid etc).

It remains to be seen, however, whether it will suffer from the same issues as gridPurchasedTodayEnergy - it may be that it too reports watts as whole numbers rather than a decimal.

+5 -1 (6 lines changed)

The value of gridPurchasedTodayEnergy is (according to gridPurchasedTodayEnergyStr) in kWh. Yet it's claiming ~650, suggesting it's actually in Wh

I'm now getting reads using the correct unit - we've pulled in 5.04kWh so I get a value of 5.04.

What's curious though, is that the behaviour didn't reproduce in the early hours of this morning. After the daily counter reset to 0, there was a period where we'd pulled hundreds of Wh from the grid, but not yet 1000 - yet the value reported by the API was correctly in kWh (i.e. it was a decimal)

Screenshot_20230601_152641

The best that I can surmise is that it's something to do with the fact that the inverter had only just been brought online yesterday. Why that should matter... but we have been getting a consistent unit today.

I've been able to build a Grafana dashboard around the stats we're collecting so far

grafana

So, I think it's probably time to look at merging the plugin into the master branch and closing this issue out - anything new can be logged as a FR/bug.

Changed have been merged and the source branch has been deleted.

mentioned in issue jira-projects/HOME#22

unknown_key

mentioned in commit github-mirror/telegraf-plugins@d8f44471a1345fc2a7ca92b5d53686be271f9a5d

Commit: github-mirror/telegraf-plugins@d8f44471a1345fc2a7ca92b5d53686be271f9a5d 
Author: Ben Tasker                            
                            
Date: 2023-06-01T15:33:02.000+01:00 

Message

Merge pull request #2 from bentasker/soliscloud-plugin

Add Soliscloud plugin (utilities/telegraf-plugins#9)

+674 -0 (674 lines changed)

mentioned in issue #10