project Utilities / Python Influxdb Downsample avatar

utilities/python_influxdb_downsample#21: Add new "copy" aggregate

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: backlog
Created: 11-Feb-23 23:40


Although not technically an aggregate, it'd be useful to add a copy mode - where points are queried from the input and queued for writing to the output.

This would allow the downsampler to be used for replicating from one source to the other.

It also means that more complex queries can be put into the query attribute, safe in the knowledge that the structure will simply be copied out to the output

Issue Links

Toggle State Changes


assigned to @btasker

It would also open the door to possibly adding support for injecting aggregateWindow - that's not desirable when running against a local install (the whole point is to use resource usage off the InfluxDB node), but might be when querying against Cloud (where the concern about resource usage doesn't exist in the same way).

mentioned in issue #22

marked this issue as related to #22

Support for this would need to go into the record loop (currently around line 457).

If the copy aggregate was active, we'd need to push an entire copy of the record into new_records.

In theory it'd be possible to have this and other aggregates active.


mentioned in commit a8540b4ad62b004de5bc1130fbdf06f29eaf4720

Commit: a8540b4ad62b004de5bc1130fbdf06f29eaf4720 
Author: B Tasker                            
Date: 2023-02-13T09:07:22.000+00:00 


fix: Convert row to a dict when copying (utilities/python_influxdb_downsample#21)

The influxdb-client doesn't support having data passed into it as a FluxRow (to be fair, it doesn't claim to) so we have to convert to a dict when queuing the point to a write.

This is more or less the same logic used later on, so to help keep things DRY, have moved out to a function

+43 -25 (68 lines changed)

mentioned in commit 0a88f6ed234e2611bd4a9788e3f1a28db01fa688

Commit: 0a88f6ed234e2611bd4a9788e3f1a28db01fa688 
Author: B Tasker                            
Date: 2023-02-13T08:55:01.000+00:00 


NotWorking: Start to implement copy aggregate utilities/python_influxdb_downsample#21

This adds code to the correct place, however, we're not currently seeing the new data at the database end - it may be that the client doesn't accept it's own format, or might be that a conditional is failing later in the process.

+13 -0 (13 lines changed)

The first commit above introduces changes in the correct place, but unfortunately the type FluxRow isn't one accepted by the write client, so the outputs weren't appearing in the database.

The second commit above introduces conversion to a dict.

The drawback of this, though, is it means that the copy aggregate cannot be used with pivoted data - it looks for _field and _value in order to determine what fields should be. Definitely worth noting in documentation.


mentioned in commit f17fe67e8486be8f6ccbcd4db4d2046e8ea80c82

Commit: f17fe67e8486be8f6ccbcd4db4d2046e8ea80c82 
Author: B Tasker                            
Date: 2023-02-13T09:18:22.000+00:00 


docs: Add details on copy aggregate (utilities/python_influxdb_downsample#21)

+12 -0 (12 lines changed)

mentioned in commit d239b46cbff4877266d808bae822f3f7f6b84e4b

Commit: d239b46cbff4877266d808bae822f3f7f6b84e4b 
Author: B Tasker                            
Date: 2023-02-13T09:34:03.000+00:00 


docs: add document on setting up replication using copy (utilities/python_influxdb_downsample#21)

+120 -0 (120 lines changed)