utilities/telegraf-plugins#12: Describe downsampling strategy

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: soliscloud-plugin vnext
Created: 09-Jun-23 16:50



Description

Figured this is as good a place as any to log this.

The Soliscloud plugin will capture data every 5 minutes or so. That's fine for looking at recent usage, but queries might get a bit painful when looking at performance over longer periods of time.

So, probably want to have data being downsampled.

The fields created vary in terms of what they represent - some are incrementing counters (so probably want downsampling with max) whilst others are current readings (so will want mean).

It might even be that we want to derive some statistics using difference and the like.



Toggle State Changes

Activity


09-Jun-23 16:50

assigned to @btasker


09-Jun-23 16:52

The easiest way to describe which field needs which is probably to build a downsample config - that's what I'm going to be using to downsample the data anyway


09-Jun-23 17:14

The inverter submits readings into Soliscloud every 5 minutes, so there's no point downsampling to a lower frequency than that.

In this case, I'm going for 30 minute samples (calculated hourly). Reducing to 1 hour feels like overkill (with a single inverter, a year of 30 minutes samples is still only 17000 points - easily downsampled further at query time if needed).

There are some stats where we only really want a mean

downsample_soliscloud_stats_mean:
    # Name for the task
    name: "Downsample Soliscloud Solar Stats (Mean)"
    influx: home1x

    # Query the last n mins
    period: 120

    # Window into n minute blocks
    window: 30

    # taken from in_bucket
    bucket: Systemstats
    measurement:
        - solar_inverter

    fields:
        - batteryHealthPerc
        - batteryPowerPerc
        - power_ac
        - state
        - temperature

    aggregates: 
        mean:

    output_influx: 
        - influx: home2xreal
    output_bucket: Systemstats/rp_720d

The incrementing counters want a max

downsample_soliscloud_stats_max:
    # Name for the task
    name: "Downsample Soliscloud Solar Stats (Max)"
    influx: home1x

    # Query the last n mins
    period: 120

    # Window into n minute blocks
    window: 30

    # taken from in_bucket
    bucket: Systemstats
    measurement:
        - solar_inverter

    fields:
        - batteryChargeToday
        - batterySupplyToday
        - batteryTodayChargeEnergy
        - batteryTodayDischargeEnergy
        - consumptionToday
        - consumptionTotal
        - gridBuyToday
        - gridBuyTotal
        - gridSellToday
        - gridSellTotal
        - localSupplyPercToday
        - localSupplyPercTotal
        - localSupplyToday
        - localSupplyTotal
        - readingAge
        - stationCapacity
        - stationCapacityUsedPerc
        - todayUsage
        - todayYield
        - totalYield

    aggregates: 
        max:

    output_influx: 
        - influx: home2xreal
    output_bucket: Systemstats/rp_720d

Whilst rate stats (like consumption) benefit from having a few different aggregates applied to them

# This job applied multiple aggregates
downsample_soliscloud_stats_multistats:
    # Name for the task
    name: "Downsample Soliscloud Solar Stats (Multiple Outputs)"
    influx: home1x

    # Query the last n mins
    period: 120

    # Window into n minute blocks
    window: 30

    # taken from in_bucket
    bucket: Systemstats
    measurement:
        - solar_inverter

    fields:
        # I only have 2 strings connected
        # so only need panel_1 and panel_2    
        - panel_1
        - panel_2
        - batteryVoltage
        - batteryCurrent
        - batteryChargeRate
        - batteryDischargeRate        
        - consumption


    aggregates: 
        min:
            field_suffix: "_min"
        max:
            field_suffix: "_max"
        mean:
        percentile:
            - 95
            - 50


    output_influx: 
        - influx: home2xreal
    output_bucket: Systemstats/rp_720d

09-Jun-23 17:18

mentioned in commit sysconfigs/downsample_configs@a06f6c8c60f6b76d380c7ffcd1d0181610bebe82

Commit: sysconfigs/downsample_configs@a06f6c8c60f6b76d380c7ffcd1d0181610bebe82 
Author: ben                            
                            
Date: 2023-06-09T18:17:08.000+01:00

Message

Add Solar downsampling jobs designed in utilities/telegraf-plugins#12

Adds 3 jobs

  • downsample_soliscloud_stats_mean
  • downsample_soliscloud_stats_max
  • downsample_soliscloud_stats_multistats
+121 -0 (121 lines changed)

09-Jun-23 17:20

The result is a bunch of fields with 30m granularity

Screenshot_20230609_181937