project Utilities / Python Influxdb Downsample avatar

utilities/python_influxdb_downsample#3: Allow multiple outputs to be specified



Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: vnext
Created: 04-Feb-23 16:38



Description

Some of my Flux Tasks still write out to multiple outputs.

It'd probably be good to be able to specify multiple outputs in a job



Toggle State Changes

Activity


assigned to @btasker

This one is probably a little more complex than it sounds.

We currently specify the output instance and bucket seperately

        output_influx: home1x
        # Which bucket?
        output_bucket: testing_db

If we're going to add support for multiple outputs, do we want to be able to specify different buckets? That'd be possible if this were a flux query as the bucket is specified when calling to().

The problem is, that quickly gets complex, with a number of failure modes.

I think the answer is probably to say that the same bucket must exist on all configured outputs.

There are a few obstacles to cross here then:

  • We need a write client per output
  • We need to maintain support for the existing string notation
  • Need to decide how to act when one write fails

Not insurmountable, but needs considering.

If we're doing this anyway, do we perhaps want to say fuck it and allow the bucket to be overidden? That'd mean accepting a dict instead of a list

        output_influx: 
             home1x:
                  bucket: testing2
        # Which bucket?
        output_bucket: testing_db

Whilst that feels a little like over-engineering, it's nothing compared to the complexity involved if we decide we later want it and end up having to support string, list and dict.

If we're doing that though, do we in fact want to accept a list of dicts, so that the result can be written out to multiple buckets on the same instance?

        output_influx: 
             - influx: home1x
               bucket: testing2
             - influx: home1x
               bucket: testing3

        # Which bucket?
        output_bucket: testing_db

Again, it's probably overengineering but leave us with the most flexibility if we want to change stuff later

verified

mentioned in commit 917e42fa9129a5f03cee07303bfdb90bf3433e71

Commit: 917e42fa9129a5f03cee07303bfdb90bf3433e71 
Author: B Tasker                            
                            
Date: 2023-02-05T14:24:01.000+00:00 

Message

docs: Update README for utilities/python_influxdb_downsample#3

+18 -0 (18 lines changed)
verified

mentioned in commit 9ce6623b1ebe4b9b2da364ce6f58b42ed04bb616

Commit: 9ce6623b1ebe4b9b2da364ce6f58b42ed04bb616 
Author: B Tasker                            
                            
Date: 2023-02-05T14:20:26.000+00:00 

Message

Implement support for multiple outputs for utilities/python_influxdb_downsample#3

This amends the attribute output_influx so that it accepts a list of objects:

output_influx:
    - influx: home1x
      bucket: testing_db/autogen2
    - influx: home1x
output_bucket: testing_db

The attribute bucket is optional, if absent the value of output_bucket will be used

+84 -9 (93 lines changed)