utilities/gadgetbridge_to_influxdb#3: Design downsampling

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: v0.4
Created: 26-Aug-23 10:03



Description

Whilst I'll probably keep a copy of the raw data for some time, it'd be worth thinking about how each of the fields can best be aggregated as part of downsampling.



Toggle State Changes

Activity


26-Aug-23 10:03

assigned to @btasker


29-Aug-23 13:20

mentioned in commit sysconfigs/downsample_configs@eb52f9406bfd2c8b64786fa269c5062a521710fe

Commit: sysconfigs/downsample_configs@eb52f9406bfd2c8b64786fa269c5062a521710fe 
Author: ben                            
                            
Date: 2023-08-29T14:19:36.000+01:00

Message

Add copy job for utilities/gadgetbridge_to_influxdb#3

This doesn't downsample, it simply copies data from the short-term DB to the _raw bucket for long term retention

+27 -0 (27 lines changed)

29-Aug-23 13:20

The job committed above is

downsample_gadgetbridge_copy_raw_data:
    # Name for the task
    name: "Copy Gadgetbridge Raw Data"
    influx: home1x

    # Query the last n mins
    period: 86400

    # Window into n minute blocks
    window: 60

    # taken from in_bucket
    bucket: telegraf

    query: >
        from(bucket: "telegraf" )
        |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
        |> filter(fn:(r) =>  r._measurement == "gadgetbridge" )

    aggregates: 
        copy:

    output_influx: 
        - influx: home2xreal
    output_bucket: health_raw

29-Aug-23 13:21

Next step then, will be to look through the fields and define which should be mean, which should be max or sum etc.

Worth keeping in mind that, unlike other data, steps is not a cumulative counter - so it'd be well worth having a sum based aggregate to calculate steps per day


30-Aug-23 07:56

Taking a quick run over then, I think the fields probably need the following aggregates applied. The assumption is that the window used will be 5 - 30 minutes. Although the same could be used for a 1d group of most of the fields, it wouldn't really work for things like heart-rate, so any daily downsamples need to be designed seperately.

Last:

  • time_low
  • time_moderate
  • time_high
  • pai_today
  • pai_total
  • last_seen_age
  • battery

Mean

  • spo2
  • pai_low
  • pai_moderate
  • pai_high

Mean & Max

  • heart_rate (sample_type=periodic_samples)
  • intensity (sample_type=periodic_samples)
  • heart_rate (sample_type=activity)
  • intensity (sample_type=activity)
  • stress_exc_sleep
  • current_stress_level (stress=point_in_time)

Sum

  • Steps (sample_type=periodic_samples)
  • Steps (sample_type=activity)
  • stress_level_counter_*

Skip

  • heart_rate (sample_type=manual, max, resting) - NOTE: Not currently supported
  • sleep_respiratory_rate - NOTE: not currently supported